[R] RAM usage
jaitchis at hwy.com.au
Sat Oct 19 09:49:32 CEST 2002
On 18 Oct 2002, at 14:03, ripley at stats.ox.ac.uk wrote:
> On Fri, 18 Oct 2002, Vaidotas Zemlys wrote:
> > I'm having problems while working with large data sets with R 1.5.1
> > in windows 2000. Given a integer matrix size of 30 columns and
> > 15000 rows my function should return a boolean matrix size of about
> > 5000 rows and 15000 columns.
> Don't think so. More likely that Windows is having problems managing
> the memory requirements. You are trying to access an object too big
> to fit into RAM, and that going to cause severe strain.
> Again, there is likely a problem with Windows allocating a contiguous
> chunk of 300Mb of memory. Try this sort of thing only after a fresh
> R can. The question is `can Windows'? If possible use a Unix-based
Windows leaves a LOT of junk lying around in RAM (recently used
DLL's etc) and even after a reboot there is still some reclaimable
RAM (from processes used in startup).
I use a program called MemTurbo http://www.memturbo.com/
which will do a ram scrub (releasing unused ram when free ram
limits are reached) and a ram defrag (so if contiguous ram is
needed then this might help). It has been around a while, I have
used it since an earlier version .. I don't believe it is perfect but it
does seem to do a good job of the arcane area of windows memory
Maybe that will help to get you "clean ram".
The other area that might be worth attention is the number of
processes that are currently running .. you can perhaps kill some
of these. And, of course defragging your hard disk(s) and perhaps
managing the swap file yourself are old favourites, not to mention
cleaning the registry .. none of these should in theory have
anything to do with memory management (by R), but in practice
there seem to be some complex "interactions" in the OS, between
the OS and the registry and RAM and concurrent threads.
I have also found that a "lightly loaded" Windows machine (one
with very few programs installed) is much more likely to be stable
than one with many programs installed, and I have a glimmering of
an idea that there is some critical size of the registry beyond which
something starts to thrash (? if the registry size is greater than
available physical RAM) . Of course none of this registry business
SHOULD affect R, but then again, Windoze is a black box, so who
knows what goes on with program loading, thread interaction etc
To cut a long story short, it might just possibly help if you try to
keep your ram and your disks and swapfiles and registry as clean
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch
More information about the R-help