[R] Memory leak in R v1.5.1?
Jan Meyer
janm at mit.edu
Tue Aug 6 18:07:07 CEST 2002
Moin Martin,
danke fuer Deine schnelle Antwort.
[I will continue in English for the sake of forwardability]
It's very understandable to me that you're asking for readable example
code. But then I was wondering, do you really want to look through ~1300
lines of code (w/o comments) generated from a collection of Sweave
files? I guess in that case I'd have to reduce my problem down to a
level that can be easily overlooked but still causes the problem.
Unfortunately I don't have the time for that during the next two weeks
(really - please don't ask).
My idea was to ask for some hints from you first, then try to fix it my
own, if that doesn't work quickly, find some ugly work around and pick
up the problem in two weeks for a real solution.
For example I don't understand how it is possible for my code to
generate this memory leak in particular, since this is not me playing
the pointer wizard in C and since it's a problem with cons cells and not
the heap. If I know what I could be doing wrong, I might be able to
solve the problem myself.
Should the consumption of cons cell ever grow during the course of a
session? [The consumption of these cells does not even go down if I
terminate the script.] I force garbage collection with gc() does that
also take care of the cons cell or heap only?
Thanks.
- Jan
Martin Maechler wrote:
>>>>>>"Jan" == Jan Meyer <janm at mit.edu> writes:
>>>>>
>
> Jan> Hi,
>
> Jan> I am trying to minimize a rather complex function of 5
> Jan> parameters with gafit and nlm. Besides some problems
> Jan> with both optimization algorithms (with respect to
> Jan> consistantly generating similar results), I tried to
> Jan> run this optimization about a hundred times for yet two
> Jan> other parameters.
>
> Jan> Unfortunately, as the log below shows, during that
> Jan> batch process R starts to eat up all my RAM, preventing
> Jan> me to simply run this job over night.
>
> Jan> The problem seems to be related to memory used by cons
> Jan> cells, which are internal data structures like parse
> Jan> trees etc. as I understand. (For more see
> Jan> memory.profile() call all the way below) Despite me
> Jan> forcing garbage collection between each optimization
> Jan> run, these cells accumulate quite rapidly.
>
> which must be a leak bug somewhere, yes,
> not necessarily in "core R" though. It could be in your code or
> in gafit() {which is from a contributed package and hence not in
> "core R"}.
>
> Jan> All I am doing is calling my function (implemented as an R function
> Jan> calling another R function) millions of times through gafit() and nlm().
> Jan> The little data that my function needs is loaded into the environment
> Jan> prior to optimization. Either algorithm seems to accumulate cons cells.
>
> Jan> Any idea what could be the cause? My code maybe? Is this a memory leak
> Jan> within R?
>
> both are possible. We have had rare cases of memory leaks
> {found and fixed} in the past.
>
> But since you mention "1.5.1" in the subject, have you used your
> code in earlier versions of R and not seen the memory growth?
>
> Jan> Is there any other way to force garbage collection (or something
> Jan> equivalent) on the ncells?
>
> Jan> Thanks.
>
> Rather we need vastly better readable example code in order to
> tell. The stuff you append looks terribly garbled (by your
> e-mail software probably?), and what we really need is
> *reproducible* source code (i.e. data, function, all available
> in that R script) rather than a session log.
> If it is true that you only have a basic function and everything
> else is nlm() and gafit() calls, such a script shouldn't be too
> long.
>
> Thank you for the report and (in advance) for such a more useful
> script.
>
> Martin Maechler <maechler at stat.math.ethz.ch> http://stat.ethz.ch/~maechler/
> Seminar fuer Statistik, ETH-Zentrum LEO C16 Leonhardstr. 27
> ETH (Federal Inst. Technology) 8092 Zurich SWITZERLAND
> phone: x-41-1-632-3408 fax: ...-1228 <><
>
>
>
> Jan> - Jan
>
> Jan> P.S. I run R v1.5.1 on i386 SuSE Linux v7.3 using the precompiled
> Jan> binaries from CTAN for SuSE v7.3.
>
> Jan> -----------------
> >> fName ="designOpt"; Sweave(paste(fName, "rnw", sep="."),
> Jan> driver=Rtangle()); sourc
> Jan> e(paste(fName, "R", sep="."), local = TRUE)
> Jan> Writing to file designOpt.R
> Jan> 4 : designOpt-designCollectionFileFuncs.R
> Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 0
> Jan> GA Design: ShipMathModel( LWL=551.205711406606, B=45.4637940845938,
> Jan> Cp=0.7198803735
> Jan> 31636, Cx=0.837682085129617, W.BP=872.396799843704,
> Jan> H.DKh=12.0780590098038, H.DKd=8
> Jan> .25941049673737 ) = (Cost=3.16 , Err.Pwr=15.0, Err.Range=18.5, B/H=3.55)
> Jan> Garbage collection 170 = 160+7+3 (level 2) ...
> Jan> 147999 cons cells free (26%)
> Jan> 7.8 Mbytes of heap free (77%)
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 417001 11.2 565000 15.1 16.1
> Jan> Vcells 291029 2.3 1310720 10.0 20.0
> Jan> total used free shared buffers cached
> Jan> Mem: 124948 93736 31212 0 17308 25824
> Jan> -/+ buffers/cache: 50604 74344
> Jan> Swap: 265032 8 265024
> Jan> [1] 0
> Jan> Gradient Design: ShipMathModel( LWL=537.059747125118,
> Jan> B=44.8171342153699, Cp=0.7222
> Jan> 67017738667, Cx=0.836634804176116, W.BP=872.197302196268,
> Jan> H.DKh=12.0780590098038, H
> Jan> .DKd=8.14724020090996 ) = (Cost=3.15 , Err.Pwr=11.5, Err.Range=25.1,
> Jan> B/H=3.45)
> Jan> Garbage collection 184 = 172+8+4 (level 2) ...
> Jan> 141287 cons cells free (25%)
> Jan> 7.7 Mbytes of heap free (77%)
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 423713 11.4 565000 15.1 16.1
> Jan> Vcells 299539 2.3 1310720 10.0 20.0
> Jan> total used free shared buffers cached
> Jan> Mem: 124948 93736 31212 0 17308 25824
> Jan> -/+ buffers/cache: 50604 74344
> Jan> Swap: 265032 8 265024
> Jan> [1] 0
> >>>>>>> im = 2 ; time = Sat Aug 3 15:37:34 2002
> Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 1.274
> Jan> GA Design: ShipMathModel( LWL=531.251923358073, B=44.9592165303293,
> Jan> Cp=0.726323984032877, Cx=0.823656217446154, W.BP=779.115652389105,
> Jan> H.DKh=11.7190083082315, H.DKd=8.55853079052759 ) = (Cost=3.16 ,
> Jan> Err.Pwr=13.7, Err.Range=14.3, B/H=3.47)
> Jan> Garbage collection 355 = 262+46+47 (level 2) ...
> Jan> 75179 cons cells free (13%)
> Jan> 7.1 Mbytes of heap free (70%)
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 489821 13.1 565000 15.1 16.1
> Jan> Vcells 383011 3.0 1310720 10.0 20.0
> Jan> total used free shared buffers cached
> Jan> Mem: 124948 93736 31212 0 17308 25824
> Jan> -/+ buffers/cache: 50604 74344
> Jan> Swap: 265032 8 265024
> Jan> [1] 0
> Jan> Gradient Design: ShipMathModel( LWL=531.076590422056,
> Jan> B=44.9466118426414, Cp=0.726049001140884, Cx=0.823594459634168,
> Jan> W.BP=779.10512867677, H.DKh=11.718702931698, H.DKd=8.55702887701944 ) =
> Jan> (Cost=3.16 , Err.Pwr=13.4, Err.Range=14.6, B/H=3.46)
> Jan> Garbage collection 373 = 268+51+54 (level 2) ...
> Jan> 70538 cons cells free (12%)
> Jan> 7.0 Mbytes of heap free (70%)
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 494462 13.3 565000 15.1 16.1
> Jan> Vcells 388922 3.0 1310720 10.0 20.0
> Jan> total used free shared buffers cached
> Jan> Mem: 124948 93736 31212 0 17308 25824
> Jan> -/+ buffers/cache: 50604 74344
> Jan> Swap: 265032 8 265024
> Jan> [1] 0
> >>>>>>> im = 3 ; time = Sat Aug 3 15:40:14 2002
> Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 2.548
> Jan> GA Design: ShipMathModel( LWL=578.21472911227, B=49.466359186738,
> Jan> Cp=0.702282574662
> Jan> H.DKd=11.5397975180451 ) = (Cost=3.25 , Err.Pwr=0.8, Err.Range=-1.5,
> Jan> B/H=3.28)
> Jan> Garbage collection 636 = 358+130+148 (level 2) ...
> Jan> 4432 cons cells free (0%)
> Jan> 6.4 Mbytes of heap free (63%)
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 560568 15.0 565000 15.1 16.1
> Jan> Vcells 472355 3.7 1310720 10.0 20.0
> Jan> total used free shared buffers cached
> Jan> Mem: 124948 94624 30324 0 17308 25824
> Jan> -/+ buffers/cache: 51492 73456
> Jan> Swap: 265032 8 265024
> Jan> [1] 0
> Jan> Error: cons memory exhausted (limit reached?)
> Jan> In addition: There were 24 warnings (use warnings() to see them)
> Jan> Lost warning messages
> >> gc()
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 563691 15.1 565000 15.1 16.1
> Jan> Vcells 476605 3.7 1310720 10.0 20.0
> >> rm(list = ls(all = TRUE))
> >> gc()
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 554338 14.9 565000 15.1 16.1
> Jan> Vcells 467858 3.6 1310720 10.0 20.0
> >> rm(list = ls(all = TRUE))
> >> gc()
> Jan> used (Mb) gc trigger (Mb) limit (Mb)
> Jan> Ncells 554338 14.9 565000 15.1 16.1
> Jan> Vcells 467858 3.6 1310720 10.0 20.0
> >> memory.profile()
> Jan> NILSXP SYMSXP LISTSXP CLOSXP ENVSXP PROMSXP
> Jan> LANGSXP
> Jan> 1 5031 264110 1401 16047 0
> Jan> 60670
> Jan> SPECIALSXP BUILTINSXP CHARSXP LGLSXP INTSXP
> Jan> 59 503 29179 1682 0 0
> Jan> 24
> Jan> REALSXP CPLXSXP STRSXP DOTSXP ANYSXP VECSXP
> Jan> EXPRSXP
> Jan> 135616 8 23982 0 0 16053
> Jan> 0
> Jan> EXTPTRSXP WEAKREFSXP
> Jan> 0 0 0
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
More information about the R-help
mailing list