[R] Why does a 2 GB RData file exceed my 16GB memory limit when reading it in?

[Jeff Newmiller]

You need more RAM to load this file. As the memory was being used in your original file, certain objects (such as numeric columns) were being shared among different higher-level objects (such as data frames). When serialized into the file those optimizations were lost, and now those columns are stored separately.

Search [1] for "shared" to learn more about measuring object memory requirements.

[1] http://adv-r.had.co.nz/memory.html

On September 2, 2020 2:31:53 PM PDT, David Jones <david.tn.jones using gmail.com> wrote:
>Thank you Uwe, John, and Bert - this is very helpful context.
>
>If it helps inform the discussion, to address John and Bert's
>questions - I actually had less memory free when I originally ran the
>analyses and saved the workspace, than when I read in the data back in
>later on (I rebooted in an attempt to free all possible memory before
>rereading the workspace back in).
>
>
>
>On Wed, Sep 2, 2020 at 1:27 PM John via R-help <r-help using
>r-project.org> wrote:
>
>>> On Wed, 2 Sep 2020 13:36:43 +0200
>>> Uwe Ligges <ligges using statistik.tu-dortmund.de> wrote:
>>>
>>> > On 02.09.2020 04:44, David Jones wrote:
>>> > > I ran a number of analyses in R and saved the workspace, which
>>> > > resulted in a 2GB .RData file. When I try to read the file back
>>> > > into R
>>> >
>>> > Compressed in RData but uncompressed in main memory....
>>> >
>>> >
>>> > > later, it won't read into R and provides the error: "Error:
>cannot
>>> > > allocate vector of size 37 Kb"
>>> > >
>>> > > This error comes after 1 minute of trying to read things in - I
>>> > > presume a single vector sends it over the memory limit. But,
>>> > > memory.limit() shows that I have access to a full 16gb of ram on
>my
>>> > > machine (12 GB are free when I try to load the RData file).
>>> >
>>> > But the data may need more....
>>> >
>>> >
>>> > > gc() shows the following after I receive this error:
>>> > >
>>> > > used (Mb) gc trigger (Mb) max used (Mb)
>>> > > Ncells 623130 33.3 4134347 220.8 5715387 305.3
>>> > > Vcells 1535682 11.8 883084810 6737.5 2100594002 16026.3
>>> >
>>> > So 16GB were used when R gave up.
>>> >
>>> > Best,
>>> > Uwe Ligges
>>>
>>> For my own part, looking at the OP's question, it does seem curious
>>> that R could write that .RData file, but on the same system not be
>able
>>> to reload something it created.  How would that work.  Wouldn't the
>>> memory limit have been exceeded BEFORE the the .RData file was
>written
>>> the FIRST time?
>>>
>>> JDougherty
>
>
>>R experts may give you a detailed explanation, but it is certainly
>possible
>>that the memory available to R when it wrote the file was different
>than
>>when it tried to read it, is it not?
>
>>Bert Gunter
>
>>"The trouble with having an open mind is that people keep coming along
>and
>>sticking things into it."
>>-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
>
>______________________________________________
>R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

-- 
Sent from my phone. Please excuse my brevity.