[R] Maximum number of tables combined with rbind

[David Winsemius]

You may be able to get more space than what you currently have, but my  
understanding is the XP in a default configuration can only address  
2.5 GB. You also need to consider that you will need open memory in  
order to do anything useful. I am not aware of limitations imposed by  
rbind per se. Tables (if you are using the R termonology) are actually  
special forms of arrays.

I just ran this test and it used about half of my 10 GB according to  
my systemmonitor:

 >  mtx <- matrix(1:(2500*150), ncol=150)
 > for (i in 1:10) { mtx <- rbind(mtx,mtx); print( 2^i) }
[1] 2
[1] 4
[1] 8
[1] 16
[1] 32
[1] 64
[1] 128
[1] 256
[1] 512
[1] 1024

Running it again only 8 times and looking at the fraction of occupied  
memory with Apple's Activity Monitor makes me think you could get 256  
of such tables into your space without too much difficulty, but the  
number could easily be twice that since my pointer overhead in a 64- 
bit system will be greater than yours. You will probably even have  
room to do something useful. The option of leaving the data in a  
database should not be overlooked if you need more space. sqldf is an  
available package:

http://code.google.com/p/sqldf/

Other informative functions include object.size() and gc(). After  
doubling that 8-fold copy of mtx to get a nine-fold doubling, I see:

 > str(mtx)
  int [1:1280000, 1:150] 1 2 3 4 5 6 7 8 9 10 ...

 > object.size(mtx)
[1] 768000200
 > gc()
            used  (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells   114618   6.2     350000   18.7    350000   18.7
Vcells 96121989 733.4  309848649 2364.0 576124419 4395.5

  Good luck;

David


On Jun 20, 2009, at 8:49 AM, gug wrote:

>
> Thanks David - you're right, "PC" is not very informative.  I am  
> using XP
> Home with 4GB, though I don't think XP accesses more than 3GB.
>
>> From following through the FAQ's and memory functions (e.g.
> "memory.limit(size = NA)") it looks like R is getting about 1535Mb  
> at the
> moment.
>
>
> David Winsemius wrote:
>>
>>> My question is: What is the maximum number of tables that can be
>>> combined
>>> with rbind?
>>> Is it driven by the number of rows in the tables, by a constraint   
>>> with
>>> the
>>> syntax of rbind itself, by my PC's memory,
>>
>> Yes, and the OS. "PC" is not a useful designation for an OS.
>>
>>> or by some other constraint?  At this point the data files I am  
>>> reading
>>> in are approx 2,500 rows x 150
>>> columns.
>>
>> That should not be any problem. On an Intel-Mac w/ 8 GB there is no
>> problem with dataframes that are 500 times that size.
>>
>> If this is on a Windows box, there is an RW-FAQ on the topic.

David Winsemius, MD
Heritage Laboratories
West Hartford, CT