[R] Computing very large distance matrix

[Roger D. Peng]

The problem is not with R; the problem is with Windows, as well 
as the x86 architecture.  You'll never really be able to access 
more than 3GB of physical memory (for a single process). 
Accessing more than 4GB requires a 64 bit processor.

-roger

Arnav Sheth wrote:
> Hello,
> 
> Thank you all for your replies.
> 
> I have just discovered that now, for some reason, R does not let me increase
> its memory limit beyond 4095 Mb.
> 
> Also, I get a different error message when computing the matrix using
> daisy() as opposed to dist():
> 
> Error: Cannot allocate vector of size 135168 Kb
> 
> Does this mean that there is no way in which I can compute this distance
> matrix on the machine I described below?
> 
> Is there any possible solution to this problem?
> 
> Thanks again to all,
> Arnav.
> 
> 
> 
> ----- Original Message -----
> From: "Prof Brian Ripley" <ripley at stats.ox.ac.uk>
> To: "Arnav Sheth" <sheth at economics.rutgers.edu>
> Cc: "R-Help" <R-help at stat.math.ethz.ch>
> Sent: Wednesday, February 25, 2004 2:12 PM
> Subject: Re: [R] Computing very large distance matrix
> 
> 
> 
>>On Wed, 25 Feb 2004, Arnav Sheth wrote:
>>
>>
>>>Hello All,
>>>
>>>I have a 131072x132 matrix for which I need to compute a regular
> 
> euclidean distance matrix, which I then need to transform and run agnes() on
> this transformed matrix. I am having trouble computing the distance matrix
> as it is fairly large and I am sure I have gone over the max.
> 
>>>The specific error I am getting is:
>>>Error in vector("double", length) : negative length vectors are not
> 
> allowed
> 
>>We've improved that message for the next release.
>>
>>
>>>I have increased the memory limit to the maximum capacity of my hard
>>>drive (which is around 20gb), with no success.
>>
>>You'd need to increase the 32-bit limit of Windows, too ....
>>
>>
>>>I am running the RGUI on Windows XP with 512 mb of RAM.
>>>
>>>Would anyone have any suggestions as to how I can overcome this problem?
>>>I would be most grateful for any help.
>>
>>No. The distance matrix is 8Gb all by itself.
>>
>>--
>>Brian D. Ripley,                  ripley at stats.ox.ac.uk
>>Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
>>University of Oxford,             Tel:  +44 1865 272861 (self)
>>1 South Parks Road,                     +44 1865 272866 (PA)
>>Oxford OX1 3TG, UK                Fax:  +44 1865 272595
>>
>>
> 
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>