[R] Error: cannot allocate vector of size x Gb (64-bit ... yet again)

[derek eder]

Hello,

I am facing the dreaded "Error: cannot allocate vector of size x Gb" and 
don't understand
enough about R (or operating system) memory management to diagnose and 
solve the problem
-- despite studying previous posts and relevant R help -- e.g.:

"Error messages beginning cannot allocate vector of size indicate a 
failure to obtain memory,
either because the size exceeded the address-space limit for a process 
or, more likely,
because the system was unable to provide the memory.
[...] On all builds of R, the maximum length (number of elements)
of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers.
In addition, the storage space cannot exceed the address limit."
- from Memory-limits {Base}


Simple question:  Given 64-bit R (AMD64 Linux) with a ulimit of 
"unlimited", can the size of an R object exceed the amount of availlable RAM
memory?

Empirically my system with 4Gb RAM and ample Swap, is failing:

 >  x <- integer(10^9)

 > object.size(x)
4000000040 bytes

 > gc()
             used   (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells    121195    6.5     350000   18.7    350000   18.7
Vcells 500124024 3815.7  606849099 4629.9 550124408 4197.2

 > matrix(x, ncol=16)
Error: cannot allocate vector of size 3.7 Gb

I don't understand how this operation violates the limits detailed in 
the Memory-limit help (above).

Thank you!


Derek Eder



-------------------------------------------------------------------------------------------------

 > version
                _
platform       x86_64-pc-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status
major          2
minor          11.1
year           2010
month          05
day            31
svn rev        52157
language       R
version.string R version 2.11.1 (2010-05-31)



derek at papanca:~$ top

top - 09:10:18 up 51 min,  4 users,  load average: 0.51, 0.51, 0.45
Tasks: 160 total,   2 running, 158 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us, 25.0%sy,  0.0%ni, 75.0%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Mem:   3796484k total,  3764852k used,    31632k free,    14204k buffers
Swap:  2929660k total,   834240k used,  2095420k free,    94800k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
  2854 derek     20   0  239m 9260 5448 S    6  0.2   0:05.53 
gnome-terminal
  1164 root      20   0  218m  31m  10m S    4  0.8   1:29.71 Xorg
  3331 derek     20   0 19276 1324  944 R    1  0.0   0:00.6  top