[R] sparse matrix, rnorm, malloc

g l glarowe at indiana.edu
Sat Jun 10 19:53:29 CEST 2006


Hi,

I'm Sorry for any cross-posting. I've reviewed the archives and could  
not find an exact answer to my question below.

I'm trying to generate very large sparse matrices (< 1% non-zero  
entries per row). I have a sparse matrix function below which works  
well until the row/col count exceeds 10,000. This is being run on a  
machine with 32G memory:

sparse_matrix <- function(dims,rnd,p) {
         ptm <- proc.time()
         x <- round(rnorm(dims*dims),rnd)
         x[((abs(x) - p) < 0)] <- 0
         y <- matrix(x,nrow=dims,ncol=dims)
         proc.time() - ptm
}

When trying to generate the matrix around 20,000 rows/cols on a  
machine with 32G of memory, the error message I receive is:

R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
Error: cannot allocate vector of size 3125000 Kb
Error in round(rnorm(dims * dims), rnd) : unable to find the argument  
'x' in selecting a method for function 'round'

* Last error line is obvious. Question:  on machine w/32G memory, why  
can't it allocate a vector of size 3125000 Kb?

When trying to generate the matrix around 30,000 rows/cols, the error  
message I receive is:

Error in rnorm(dims * dims) : cannot allocate vector of length 900000000
Error in round(rnorm(dims * dims), rnd) : unable to find the argument  
'x' in selecting a method for function 'round'

* Last error line is obvious. Question: is this 900000000 bytes?  
kilobytes? This error seems to be specific now to rnorm, but it  
doesn't indicate the length metric (b/Kb/Mb) as it did for 20,000  
rows/cols. Even if this Mb, why can't this be allocated on a machine  
with 32G free memory?

When trying to generate the matrix with over 50,000 rows/cols, the  
error message I receive is:

Error in rnorm(n, mean, sd) : invalid arguments
In addition: Warning message:
NAs introduced by coercion
Error in round(rnorm(dims * dims), rnd) : unable to find the argument  
'x' in selecting a method for function 'round'

* Same.

Why would it generate different errors in each case? Code fixes? Any  
simple ways to generate sparse matrices which would avoid above  
problems?

Thanks in advance,

Gavin



More information about the R-help mailing list