[R] Large file size while persisting rpart model to disk

[Duncan Murdoch]

One correction below, and a suggested alternative approach.

On 2/4/2009 9:31 AM, Terry Therneau wrote:
>   In R, functions remember their entire calling chain.  The good thing about 
> this is that they can find variables further up in the nested context, i.e.,
>     mfun <- function(x) { x+y}
> will look for 'y' in the function that called myfun, then in the function that
> called the function, .... on up and then through the search() list.  This makes
> life easier for certain things such as minimizers.

This description is not right: it's not the caller, it's the environment 
where mfun was created.  So it applies to nested functions (as you 
said), but the caller is irrelevant.

> 
>   The bad thing is that to make this work R has to remember all of the variables 
> that were available up the entire chain, and 99-100% of them aren't necessary.  
> (Because of constructs like get(varname) a parser can't read the code to decide 
> what might be needed).  

I'm not sure what you mean by "chain" here, but the real issue is that 
all the variables in the function that creates mfun will be kept as long 
as mfun exists.

> 
>   This is an issue with embedded functions.  I recently noticed an extreme case 
> of it in the pspline routine and made changes to fix it.  The short version
>   	pspline(x, ...other args) {
>   		some computations to define an X matrix, which can be large
>   		define a print function
>   		...
>   		return(X, printfun, other stuff)
>   		}

So here printfun captures all the local variables in pspline, even if it 
doesn't need them.

> It's even worse in the frailty functions, where X can be VERY large.
> The print function's environment wanted to 'remember' all of the temporary work 
> that went into defining X, plus X itself and so would be huge.  My solution was 
> add the line
> 	environment(printfun) <- new.env(parent=baseenv())
> which marks the function as not needing anything from the local environment, 
> only the base R definitions.  This would probably be a good addition to rpart, 
> but I need to look closer.
>    My first cut was to use emptyenv(), but that wasn't so smart.  It leaves 
> everything undefined, like "+" for instance. :-)

Another approach is simply to rm() the variables that aren't needed 
before returning a function.  For example, this function has locals x 
and y, but only needs y for the returned function to work:

 > fnbuilder <- function(n) {
+    x <- numeric(n)
+    y <- numeric(n)
+    noneedforx <- function() sum(y)
+    rm(x)
+    return(noneedforx)
+ }
 > f <- fnbuilder(10000)
 > f()
[1] 0

To see what actually got carried along with f, use ls():

 > ls(environment(f))
[1] "n"          "noneedforx" "y"

So we've picked up the arg n, and our local copy of noneedforx, but we 
did manage to get rid of x.  (The local copy costs almost nothing:  R 
will just have another reference to the same object as f refers to.  The 
arg could have been rm'd too, if it was big enough to matter.)

Duncan Murdoch

>    
>    	Terry Therneau
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.