[R] reading big data frame

herodote@oreka.com herodote at oreka.com
Wed Oct 26 16:01:06 CEST 2005


hy all,

I'm under linux,
I have now a big file (45 méga bytes), this file is a line of header defining the names of the columns, the rows are only integers and there is 75 columns (and 239096 rows included the header).

Before working with this big file i was working with another one (1.7 méga bytes) , loading it with read.table("data.dat",h=T) (i've tested too with scan("data.dat",what="integer",skip=1)), there where no problems, it where taken approx 1 sec to load it into R.

but now with the big file R it seems that R stay stuck (i've wait more than 5 minutes and R don't come back from the read.table function, i've tested with scan("data.dat",what="integer",skip=1)). 

I say it stay stuck because i've spy R with the linux "top" command line, and R begin to take all my memory (no prob with that) and 100% of my cpu (no prob too), then after 3 or 4 minutes it take only 3% of cpu and the memory allocated to R is average 90%, but R never come back from these functions (or i haven't wait enough, and i need to give a result as speed as i can unfortunatly).

I've divided the size of the file by 2 , then it works (i've only tested with scan) but it is too slow...

Are there some solutions to load it in one piece?
Or have i to truncate the file and access it piece by piece?


thks all
guillaume.




More information about the R-help mailing list