[Rd] astonishing memory phenomenon

[Prof Brian Ripley]

> From: "Jens Oehlschlägel-Akiyoshi" <jens.oehlschlaegel-akiyoshi@mdfactory.de>
> Date: Fri, 11 Feb 2000 13:15:03 +0100
> 
> I have a question concerning memory.
> 
> I understood that R takes a fixed amount of memory at startup (which I can
> influence with --vsize --nsize) and that gc() shows the memory still free of
> the total memory reserved for R.
> 
> However, if I create a long vector of character data, gc() only seem to
> reflect the space needed for a vector of pointers to char, the space used
> for the character data itself is ... hidden!?

You only have one character data object, of length 0, 4096, 8192,
and lots of references to it.  You don't need multiple copies of the same
data.

Try this again with a character vector of different strings.

> x <- as.character(1:1000); gc(); rm(x); gc()
         free  total (Mb)
Ncells 271221 400000  7.7
Vcells 743268 786432  6.0
         free  total (Mb)
Ncells 272223 400000  7.7
Vcells 745769 786432  6.0
> x <- as.character(1:100000); gc(); rm(x); gc()
         free  total (Mb)
Ncells 172221 400000  7.7
Vcells 495768 786432  6.0
         free  total (Mb)
Ncells 272223 400000  7.7
Vcells 745769 786432  6.0



> Now character data, gc() tells us, that we need half that much RAM as
> compared to double numbers.
> for empty character WHATEVER SIZE THE CHARACTER DATA HAVE
> 
> > x <- rep(paste(rep("0", 0), collapse=""), 700000); gc(); rm(x); gc()
>          free  total (Mb)
> Ncells  61036 250000  4.8
> Vcells 350955 786432  6.0
>          free  total (Mb)
> Ncells  61039 250000  4.8
> Vcells 700958 786432  6.0
> 
> > x <- rep(paste(rep("0", 4096), collapse=""), 700000); gc(); rm(x); gc()
>          free  total (Mb)
> Ncells  61033 250000  4.8
> Vcells 350441 786432  6.0
>          free  total (Mb)
> Ncells  61036 250000  4.8
> Vcells 700956 786432  6.0
> 
> 
> > x <- rep(paste(rep("0", 8192), collapse=""), 700000); gc(); rm(x); gc()
> Error: heap memory (6144 Kb) exhausted [needed 2734 Kb more]
>        See "help(Memory)" on how to increase the heap size.
> 
> 
> Furthermore the last x successfully  created should contain data of size
> 700000*4096 Bytes corresponding to  700000*4096/1024/1024 = 2734 MB, which
> is impossible given the startup memory, impossible given RAM, impossible
> given that no swaping occurs, furthermore the OS (WinNT4) never shows any
> additional memory eaten up except that taken at startup.
> 
> What's going on?

Object sharing, and more generally a copy-if-changed strategy.

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._