[R] 64-bit R on 64-bit Windows box... Still not enough memory?!

[Philipp Pagel]

On Thu, Aug 05, 2010 at 04:40:48PM -0700, noclue_ wrote:
> 
> I have a 64-bit windows box -
>     Intel Xeon CPU E7340 @ 2.4GHz 31.9GB of RAM
> I have R 2.11.1 (64bit) running on it.
> 
> My csv data is 3.6 GB (with about 15 million obs, 120 variables.)

Here is my guess: Your vraiables are mstly numeric but only given with
two significant digits in the csv file:

  A     B     ...
0.0  12.0
1.3   0.4
2.3   1.1

So that would make

15e6 * 120 * 3 / 1024^3 = 5.0 Gb

You ahve 3.6Gb - but that's close enough. If you read that into R,
each nume ris represented as a double - i.e. 8 byte. Thus the entire
data frame takes

15e6 * 120 * 8 / 1024^3 = 13.4Gb

With almost half of your memory taken things can get problematic. Once
you start actually working with the data you'll have to allow for a
lot more space because copies will probably be made in the process.

So you may have to put your data into a database and process it in
pieces. Or use sqldf or bigmemory or something like that.

cu
	Philipp

-- 
Dr. Philipp Pagel
Lehrstuhl für Genomorientierte Bioinformatik
Technische Universität München
Wissenschaftszentrum Weihenstephan
Maximus-von-Imhof-Forum 3
85354 Freising, Germany
http://webclu.bio.wzw.tum.de/~pagel/