[R] Memory leak in R v1.5.1?
Martin Maechler
maechler at stat.math.ethz.ch
Tue Aug 6 09:39:46 CEST 2002
>>>>> "Jan" == Jan Meyer <janm at mit.edu> writes:
Jan> Hi,
Jan> I am trying to minimize a rather complex function of 5
Jan> parameters with gafit and nlm. Besides some problems
Jan> with both optimization algorithms (with respect to
Jan> consistantly generating similar results), I tried to
Jan> run this optimization about a hundred times for yet two
Jan> other parameters.
Jan> Unfortunately, as the log below shows, during that
Jan> batch process R starts to eat up all my RAM, preventing
Jan> me to simply run this job over night.
Jan> The problem seems to be related to memory used by cons
Jan> cells, which are internal data structures like parse
Jan> trees etc. as I understand. (For more see
Jan> memory.profile() call all the way below) Despite me
Jan> forcing garbage collection between each optimization
Jan> run, these cells accumulate quite rapidly.
which must be a leak bug somewhere, yes,
not necessarily in "core R" though. It could be in your code or
in gafit() {which is from a contributed package and hence not in
"core R"}.
Jan> All I am doing is calling my function (implemented as an R function
Jan> calling another R function) millions of times through gafit() and nlm().
Jan> The little data that my function needs is loaded into the environment
Jan> prior to optimization. Either algorithm seems to accumulate cons cells.
Jan> Any idea what could be the cause? My code maybe? Is this a memory leak
Jan> within R?
both are possible. We have had rare cases of memory leaks
{found and fixed} in the past.
But since you mention "1.5.1" in the subject, have you used your
code in earlier versions of R and not seen the memory growth?
Jan> Is there any other way to force garbage collection (or something
Jan> equivalent) on the ncells?
Jan> Thanks.
Rather we need vastly better readable example code in order to
tell. The stuff you append looks terribly garbled (by your
e-mail software probably?), and what we really need is
*reproducible* source code (i.e. data, function, all available
in that R script) rather than a session log.
If it is true that you only have a basic function and everything
else is nlm() and gafit() calls, such a script shouldn't be too
long.
Thank you for the report and (in advance) for such a more useful
script.
Martin Maechler <maechler at stat.math.ethz.ch> http://stat.ethz.ch/~maechler/
Seminar fuer Statistik, ETH-Zentrum LEO C16 Leonhardstr. 27
ETH (Federal Inst. Technology) 8092 Zurich SWITZERLAND
phone: x-41-1-632-3408 fax: ...-1228 <><
Jan> - Jan
Jan> P.S. I run R v1.5.1 on i386 SuSE Linux v7.3 using the precompiled
Jan> binaries from CTAN for SuSE v7.3.
Jan> -----------------
>> fName ="designOpt"; Sweave(paste(fName, "rnw", sep="."),
Jan> driver=Rtangle()); sourc
Jan> e(paste(fName, "R", sep="."), local = TRUE)
Jan> Writing to file designOpt.R
Jan> 4 : designOpt-designCollectionFileFuncs.R
Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 0
Jan> GA Design: ShipMathModel( LWL=551.205711406606, B=45.4637940845938,
Jan> Cp=0.7198803735
Jan> 31636, Cx=0.837682085129617, W.BP=872.396799843704,
Jan> H.DKh=12.0780590098038, H.DKd=8
Jan> .25941049673737 ) = (Cost=3.16 , Err.Pwr=15.0, Err.Range=18.5, B/H=3.55)
Jan> Garbage collection 170 = 160+7+3 (level 2) ...
Jan> 147999 cons cells free (26%)
Jan> 7.8 Mbytes of heap free (77%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 417001 11.2 565000 15.1 16.1
Jan> Vcells 291029 2.3 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
Jan> Gradient Design: ShipMathModel( LWL=537.059747125118,
Jan> B=44.8171342153699, Cp=0.7222
Jan> 67017738667, Cx=0.836634804176116, W.BP=872.197302196268,
Jan> H.DKh=12.0780590098038, H
Jan> .DKd=8.14724020090996 ) = (Cost=3.15 , Err.Pwr=11.5, Err.Range=25.1,
Jan> B/H=3.45)
Jan> Garbage collection 184 = 172+8+4 (level 2) ...
Jan> 141287 cons cells free (25%)
Jan> 7.7 Mbytes of heap free (77%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 423713 11.4 565000 15.1 16.1
Jan> Vcells 299539 2.3 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
>>>>>>> im = 2 ; time = Sat Aug 3 15:37:34 2002
Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 1.274
Jan> GA Design: ShipMathModel( LWL=531.251923358073, B=44.9592165303293,
Jan> Cp=0.726323984032877, Cx=0.823656217446154, W.BP=779.115652389105,
Jan> H.DKh=11.7190083082315, H.DKd=8.55853079052759 ) = (Cost=3.16 ,
Jan> Err.Pwr=13.7, Err.Range=14.3, B/H=3.47)
Jan> Garbage collection 355 = 262+46+47 (level 2) ...
Jan> 75179 cons cells free (13%)
Jan> 7.1 Mbytes of heap free (70%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 489821 13.1 565000 15.1 16.1
Jan> Vcells 383011 3.0 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
Jan> Gradient Design: ShipMathModel( LWL=531.076590422056,
Jan> B=44.9466118426414, Cp=0.726049001140884, Cx=0.823594459634168,
Jan> W.BP=779.10512867677, H.DKh=11.718702931698, H.DKd=8.55702887701944 ) =
Jan> (Cost=3.16 , Err.Pwr=13.4, Err.Range=14.6, B/H=3.46)
Jan> Garbage collection 373 = 268+51+54 (level 2) ...
Jan> 70538 cons cells free (12%)
Jan> 7.0 Mbytes of heap free (70%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 494462 13.3 565000 15.1 16.1
Jan> Vcells 388922 3.0 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
>>>>>>> im = 3 ; time = Sat Aug 3 15:40:14 2002
Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 2.548
Jan> GA Design: ShipMathModel( LWL=578.21472911227, B=49.466359186738,
Jan> Cp=0.702282574662
Jan> H.DKd=11.5397975180451 ) = (Cost=3.25 , Err.Pwr=0.8, Err.Range=-1.5,
Jan> B/H=3.28)
Jan> Garbage collection 636 = 358+130+148 (level 2) ...
Jan> 4432 cons cells free (0%)
Jan> 6.4 Mbytes of heap free (63%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 560568 15.0 565000 15.1 16.1
Jan> Vcells 472355 3.7 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 94624 30324 0 17308 25824
Jan> -/+ buffers/cache: 51492 73456
Jan> Swap: 265032 8 265024
Jan> [1] 0
Jan> Error: cons memory exhausted (limit reached?)
Jan> In addition: There were 24 warnings (use warnings() to see them)
Jan> Lost warning messages
>> gc()
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 563691 15.1 565000 15.1 16.1
Jan> Vcells 476605 3.7 1310720 10.0 20.0
>> rm(list = ls(all = TRUE))
>> gc()
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 554338 14.9 565000 15.1 16.1
Jan> Vcells 467858 3.6 1310720 10.0 20.0
>> rm(list = ls(all = TRUE))
>> gc()
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 554338 14.9 565000 15.1 16.1
Jan> Vcells 467858 3.6 1310720 10.0 20.0
>> memory.profile()
Jan> NILSXP SYMSXP LISTSXP CLOSXP ENVSXP PROMSXP
Jan> LANGSXP
Jan> 1 5031 264110 1401 16047 0
Jan> 60670
Jan> SPECIALSXP BUILTINSXP CHARSXP LGLSXP INTSXP
Jan> 59 503 29179 1682 0 0
Jan> 24
Jan> REALSXP CPLXSXP STRSXP DOTSXP ANYSXP VECSXP
Jan> EXPRSXP
Jan> 135616 8 23982 0 0 16053
Jan> 0
Jan> EXTPTRSXP WEAKREFSXP
Jan> 0 0 0
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
More information about the R-help
mailing list