[R] Memory filling up while looping

[Peter Meißner]

I'll consider it. But in fact the whole data does not fit into memory at 
once with the overhead to create it in addition - I think. That was one 
of the reasons I wanted to do it chunk by chunk in the first place.

Thanks, Best, Peter

Am 21.12.2012 15:07, schrieb Duncan Murdoch:
> On 12-12-20 6:26 PM, Peter Meissner wrote:
>> Hey,
>>
>> I have an double loop like this:
>>
>>
>> chunk <- list(1:10, 11:20, 21:30)
>> for(k in 1:length(chunk)){
>>     print(chunk[k])
>>     DummyCatcher <- NULL
>>     for(i in chunk[k]){
>>         print("i load something")
>>         dummy <- 1
>>         print("i do something")
>>         dummy <- dummy + 1
>>         print("i do put it together")
>>         DummyCatcher = rbind(DummyCatcher, dummy)
>>     }
>>     print("i save a chunk and restart with another chunk of data")
>> }
>>
>> The problem now is that with each 'chunk'-cycle the memory used by R
>> becomes bigger and bigger until it exceeds my RAM but the RAM it needs
>> for any of the chunk-cycles alone is only a 1/5th of what I have overall.
>>
>> Does somebody have an idea why this behaviour might occur? Note that all
>> the objects (like 'DummyCatcher') are reused every cycle so that I would
>> assume that the RAM used should stay about the same after the first
>> 'chunk' cycle.
>
> You should pre-allocate your result matrix.  By growing it a few rows at
> a time, R needs to do this:
>
> allocate it
> allocate a bigger one, copy the old one in
> delete the old one, leaving a small hole in memory
> allocate a bigger one, copy the old one in
> delete the old one, leaving a bigger hold in memory, but still too small
> to use...
>
> etc.
>
> If you are lucky, R might be able to combine some of those small holes
> into a bigger one and use that, but chances are other variables will
> have been created there in the meantime, so the holes will go mostly
> unused.  R never moves an object during garbage collection, so if you
> have fragmented memory, it's mostly wasted.
>
> If you don't know how big the final result will be, then allocate large,
> and when you run out, allocate bigger.  Not as good as one allocation,
> but better than hundreds.
>
> Duncan Murdoch
>
>>
>>
>> Best, Peter
>>
>>
>> SystemInfo:
>>
>> R version 2.15.2 (2012-10-26)
>> Platform: x86_64-w64-mingw32/x64 (64-bit)
>> Win7 Enterprise, 8 GB RAM
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>

-- 
Peter Meißner
Workgroup 'Comparative Parliamentary Politics'
Department of Politics and Administration
University of Konstanz
Box 216
78457 Konstanz
Germany

+49 7531 88 5665
http://www.polver.uni-konstanz.de/sieberer/home/