[R] How to read HUGE data sets?

[Patrick Connolly]

On Wed, 27-Feb-2008 at 09:13PM -0800, Roy Mendelssohn wrote:

|> I may be mistaken, but I believe R does all it work in memory.  If  
|> that is so, you would really only have 2 options:
|> 
|> 1.  Get a lot of memory

But with a 32bit operating system, 4G is all the memory that can be
addressed (including the operating system).  So your chances of
getting all the data into R seem very slim.

|> 
|> 2.  Figure out a way to do the desired operation on parts of the data  
|> at a time.

That might involve using a database which you can query from R, or you
might be able to use a Perl script to select what you require.  I have
heard of people using Perl with Windows.


Someone once asked me to plot some SAS output which was several
hundred Mb.  In that case, a simple Perl script cut it down to 3 Mb.
You might be lucky too.


Good luck.

-- 
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.   
   ___    Patrick Connolly   
 {~._.~}          		 Great minds discuss ideas    
 _( Y )_  	  	        Middle minds discuss events 
(:_~*~_:) 	       		 Small minds discuss people  
 (_)-(_)  	                           ..... Anon
	  
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.