[R] load huge image

[Henrik Bengtsson]

On 3/27/06, Martin Maechler <maechler at stat.math.ethz.ch> wrote:
> >>>>> "Gottfried" == Gottfried Gruber <ggruber at terminal.at>
> >>>>>     on Sun, 26 Mar 2006 10:27:35 +0200 writes:
>
>     Gottfried> hello, i have run around 65000 regressions and
>     Gottfried> stored them in a list. then i stored the session
>     Gottfried> with save.image on my hard disk. the file is
>     Gottfried> almost 1GB. when i now want to load the image it
>     Gottfried> took tons of time. even after 12h of loading it
>     Gottfried> was not done, although the saving was done fairly
>     Gottfried> fast.
>
> I'm sure it takes so lang because you (i.e. R) run out of RAM
> and the machine starts to swap.
>
> Try to get access to a Linux (or other Unix-alike) machine with
> a 64-bit version of R and about 8 GB of RAM (maybe 4 GB is
> already sufficient). I guess then you should be able to read it
> much more quickly.
>
> For 65000 regressions, do you need more than the estimated
> coefficients or -- a bit more informatively -- the
>
>   coef(summary( <lm-fit> ))  result ?
>
> If you had only saved these coefficient matrices, I'm sure you'd
> have need **much** less memory.
>
>
>     Gottfried> i fear i have to run the regressions again and
>     Gottfried> store them in a database ...
>
> or really store what you need instead of everything ...
>
>     Gottfried> can i load this file? any suggestions?

Do you need to have all of them in memory at once?  Instead of using
save.image() can't you use save()/load() on each of the regression
fits?  You can name the files using sprintf("regression%05d.Rdata",
idx) or similar.

Also, as Martin says, "fit" objects contains a lot of information that
you might not need; remove these before saving by setting the elements
you don't want to NULL.

/Henrik

>     Gottfried> thanks & bets regards, gg --
>     Gottfried> ---------------------------------------------------
>     Gottfried> Gottfried Gruber
>     Gottfried> mailto:gottfried.gruber at terminal.at www:
>     Gottfried> http://gogo.sehrsupa.net
>
>     Gottfried> ______________________________________________
>     Gottfried> R-help at stat.math.ethz.ch mailing list
>     Gottfried> https://stat.ethz.ch/mailman/listinfo/r-help
>     Gottfried> PLEASE do read the posting guide!
>     Gottfried> http://www.R-project.org/posting-guide.html
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>
>


--
Henrik Bengtsson
Mobile: +46 708 909208 (+2h UTC)