[R] Why does a 2 GB RData file exceed my 16GB memory limit when reading it in?

Ista Zahn |@t@z@hn @end|ng |rom gm@||@com
Thu Sep 3 15:54:47 CEST 2020


On Wed, Sep 2, 2020 at 7:22 PM Leandro Marino
<leandromarino using leandromarino.com.br> wrote:
>
> David,
>
> If the ".Rdata" contains more than one object you could (and maybe should
> use) the SOAR package (from Venables). This package helps you to split the
> objects over multiple RData files. It's useful when you have numerous
> medium-large objects in the workspace but doesn't use then at the same
> time.
>
> When use SOAR::Attach(), for instance, it loads the current name of all the
> objects and retain than available in the searchpath but without load then
> to the memory. As you call, they will be loaded into the memory.
>
> If needed, you can update the object and then store it again with the
> SOAR::Store()
>
> For my use, this package is terrific! I use it with an analysis that I need
> to repeat over medium-large similars datasets.
>

The qs package might also be worth a try. I don't have a specific
reason for thinking it will avoid the original problem, but in general
qs uses lots of fancy compression and memory management features.

--Ista

> Best
> Leandro
>
> Em qua., 2 de set. de 2020 às 18:33, David Jones <david.tn.jones using gmail.com>
> escreveu:
>
> > Thank you Uwe, John, and Bert - this is very helpful context.
> >
> > If it helps inform the discussion, to address John and Bert's
> > questions - I actually had less memory free when I originally ran the
> > analyses and saved the workspace, than when I read in the data back in
> > later on (I rebooted in an attempt to free all possible memory before
> > rereading the workspace back in).
> >
> >
> >
> > On Wed, Sep 2, 2020 at 1:27 PM John via R-help <r-help using
> > r-project.org> wrote:
> >
> > >> On Wed, 2 Sep 2020 13:36:43 +0200
> > >> Uwe Ligges <ligges using statistik.tu-dortmund.de> wrote:
> > >>
> > >> > On 02.09.2020 04:44, David Jones wrote:
> > >> > > I ran a number of analyses in R and saved the workspace, which
> > >> > > resulted in a 2GB .RData file. When I try to read the file back
> > >> > > into R
> > >> >
> > >> > Compressed in RData but uncompressed in main memory....
> > >> >
> > >> >
> > >> > > later, it won't read into R and provides the error: "Error: cannot
> > >> > > allocate vector of size 37 Kb"
> > >> > >
> > >> > > This error comes after 1 minute of trying to read things in - I
> > >> > > presume a single vector sends it over the memory limit. But,
> > >> > > memory.limit() shows that I have access to a full 16gb of ram on my
> > >> > > machine (12 GB are free when I try to load the RData file).
> > >> >
> > >> > But the data may need more....
> > >> >
> > >> >
> > >> > > gc() shows the following after I receive this error:
> > >> > >
> > >> > > used (Mb) gc trigger (Mb) max used (Mb)
> > >> > > Ncells 623130 33.3 4134347 220.8 5715387 305.3
> > >> > > Vcells 1535682 11.8 883084810 6737.5 2100594002 16026.3
> > >> >
> > >> > So 16GB were used when R gave up.
> > >> >
> > >> > Best,
> > >> > Uwe Ligges
> > >>
> > >> For my own part, looking at the OP's question, it does seem curious
> > >> that R could write that .RData file, but on the same system not be able
> > >> to reload something it created.  How would that work.  Wouldn't the
> > >> memory limit have been exceeded BEFORE the the .RData file was written
> > >> the FIRST time?
> > >>
> > >> JDougherty
> >
> >
> > >R experts may give you a detailed explanation, but it is certainly
> > possible
> > >that the memory available to R when it wrote the file was different than
> > >when it tried to read it, is it not?
> >
> > >Bert Gunter
> >
> > >"The trouble with having an open mind is that people keep coming along and
> > >sticking things into it."
> > >-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
> >
> > ______________________________________________
> > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.



More information about the R-help mailing list