[Rd] gc()$Vcells < 0 (PR#9345)

[Prof Brian Ripley]

On Mon, 6 Nov 2006, Vladimir Dergachev wrote:

> On Monday 06 November 2006 6:12 pm, dmaszle at mendelbio.com wrote:
>> version.string Version 2.3.0 (2006-04-24)
>>
>>> x<-matrix(nrow=44000,ncol=48000)
>>> y<-matrix(nrow=44000,ncol=48000)
>>> z<-matrix(nrow=44000,ncol=48000)
>>> gc()
>>
>>               used    (Mb) gc trigger    (Mb) max used    (Mb)
>> Ncells      177801     9.5     407500    21.8   350000    18.7
>> Vcells -1126881981 24170.6         NA 24173.4       NA 24170.6
>>
>
> Happens to me with versions 2.40 and 2.3.1. The culprit is this line
> in src/main/memory.c:
>
>    INTEGER(value)[1] = R_VSize - VHEAP_FREE();
>
> Since the amount used is greater than 4G and INTEGER is 32bit long
> (even on 64 bit machines) this returns (harmless) nonsense.

That's not quite correct.  The units here are Vcells (8 bytes), and 
integer() is signed, so this can happen only if more than 16Gb of heap is 
allocated.

We are aware that we begin to hit problems at 16Gb: it is for example the 
maximum size of an R vector.  Those objects are logical and so about 7.8Gb 
each: their length as vectors is 98% of the maximum possible.  However, 
the first time we discussed it we thought it would be about 5 years before 
those limits would become important -- I think three of those years have 
since passed.

> The megabyte value nearby is correct and gc trigger and max used fields are
> marked as NA already.

and now 'used' is also marked as NA in 2.4.0 patched.

This is only a reporting issue.  When I first used R it reported only 
numbers, and I added the Mb as a more comprehensible figure (especially 
for Ncells).  I think it would be sensible now to only report these 
figures in Mb or Gb (and also the reports for gcinfo(TRUE)).

The model behind the report actually pre-dates the GC change in 1.2.0. 
The 'Vcells' are nowadays the sum of all the allocations from VECSXPs 
(which include their headers), rather than the 'vector heap' (although 
some of the earlier terminology persists).

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595