[R] RAM usage

[John Aitchison]

On 18 Oct 2002, at 14:03, ripley at stats.ox.ac.uk wrote:

> On Fri, 18 Oct 2002, Vaidotas Zemlys wrote:
> 
> > I'm having problems while working with large data sets with R 1.5.1
> > in windows 2000. Given a integer matrix size of 30 columns and 
> > 15000 rows my function should return a boolean matrix size of about
> > 5000 rows and 15000 columns.

snip
> 
> 
> Don't think so.  More likely that Windows is having problems managing
> the memory requirements.  You are trying to access an object too big
> to fit into RAM, and that going to cause severe strain.

snip
> 
> Again, there is likely a problem with Windows allocating a contiguous
> chunk of 300Mb of memory.  Try this sort of thing only after a fresh
> reboot.
> 
snip

> 
> R can. The question is `can Windows'?  If possible use a Unix-based
> OS.

Windows leaves a LOT of junk lying around in RAM (recently used 
DLL's etc)  and even after a reboot there is still some reclaimable 
RAM (from processes used in startup). 

I use a program called MemTurbo http://www.memturbo.com/
which will do a ram scrub (releasing unused ram when free ram 
limits are reached) and a ram defrag (so if contiguous ram is 
needed then this might help). It has been around a while, I have 
used it since an earlier version .. I don't believe it is perfect but it 
does seem to do a good job of the arcane area of windows memory 
management. 

Maybe that will help to get you "clean ram".

The other area that might be worth attention is the number of 
processes that are currently running .. you can perhaps kill some 
of these. And, of course defragging your hard disk(s) and perhaps 
managing the swap file yourself are old favourites, not to mention 
cleaning the registry .. none of these should in theory have 
anything to do with memory management (by R), but in practice 
there seem to be some complex "interactions" in the OS, between 
the OS and the registry and RAM and concurrent threads. 

I have also found that a "lightly loaded"  Windows machine (one 
with very few programs installed) is much more likely to be stable 
than one with many programs installed, and I have a glimmering of 
an idea that there is some critical size of the registry beyond which 
something starts to thrash (? if the registry size is greater than 
available physical RAM) . Of course none of this registry business  
SHOULD affect R, but then again, Windoze is a black box, so who 
knows what goes on with program loading, thread interaction etc 
etc


To cut a long story short, it might just possibly help if you try to 
keep your ram and your disks and swapfiles and registry as clean 
as possible.


fwiw





-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._