[R] Large 3d array manipulation

[David Winsemius]

On Feb 27, 2009, at 7:49 PM, Duncan Murdoch wrote:

> On 27/02/2009 6:15 PM, Vemuri, Aparna wrote:
>> I have a large 3 dimensional array of size (243,246,768)
>> The first dimension is Rows, second is columns and the third is  
>> Time. So for each row and column, I want to calculate the mean of  
>> time steps
>> 1:8, 2:9, 3:10 and so on and assign the values to a new array. For  
>> this
>> I am using the following script.
>> for(i in 1:243)
>> {
>> for(j in 1:246)
>> {
>> for(k in 1:768)
>> {
>> newVar[i,j,k] <- mean( myVar[i,j,k:k+7])
>> }
>> }
>> }
>> This works, but needless to mention it take a very long time to loop
>> over all the rows, columns and time periods. I was wondering if  
>> there is
>> a simpler way to do this.
>
> Yes, vectorize it.  I think this would work:
>
> newVar <- array(NA, c(243, 246, 768))
>
> for (k in 1:768)
>  newVar[,,k] <- apply(myVar, 1:2, function(x) mean(x[k:(k+7)]))

That's rather interesting. I had not realized that one could use that  
construction with apply. I had earlier tried to substitute rollmean  
for the inner loop. For one thing I thought that trying to index 768+7  
was going to create some problems with out-of-range indexing. I got  
the same error with my effort to insert rollmean in the OP's  
construction as I do in this construction:

myVar <- array(1:(243*246*768), dim=c(243,246,768))
 > myVar[1,1,1:8]
[1]      1  59779 119557 179335 239113 298891 358669 418447

newVar <-array(,dim=c(243,246,768))
library(zoo)
for (k in 1:768)  newVar[,,k] <- apply(myVar, 1:2, function(x)  
mean(x[k:(k+7)]))
Error in newVar[, , k] <- apply(myVar, 1:2, function(x) rollmean(x,  
8)) :
   number of items to replace is not a multiple of replacement length

I am guessing that at some point the assignment function cannot  
resolve which index in newVar to use. So I tried redimensioning newVar  
to only have 761 as it third dimension and using:

  myVar[1,1,1:8]

for (k in 1:761)  {newVar[ , , ] <- apply(myVar, 1:2, function(x)  
mean(x[k:(k+7)])) ; print(k)}

This may be working. This executes in less than 20 seconds:

 > newVar <-array(,dim=c(243,246,761))
 > str(newVar); Sys.time()
  logi [1:243, 1:246, 1:761] NA NA NA NA NA NA ...
[1] "2009-02-27 22:35:48 EST"
 >  newVar[,,] <- apply(myVar, 1:2, function(x) rollmean(x, 8));  
Sys.time()
[1] "2009-02-27 22:35:57 EST"
 > str(newVar)
  num [1:243, 1:246, 1:761] 209224 269002 328780 388558 448336 ...
 >

-- 
David Winsemius
Heritage Laboratories