[R] Large 3d array manipulation

[David Winsemius]

On Feb 27, 2009, at 10:37 PM, David Winsemius wrote:

>
> On Feb 27, 2009, at 7:49 PM, Duncan Murdoch wrote:
>
>> On 27/02/2009 6:15 PM, Vemuri, Aparna wrote:
>>> I have a large 3 dimensional array of size (243,246,768)
>>> The first dimension is Rows, second is columns and the third is  
>>> Time. So for each row and column, I want to calculate the mean of  
>>> time steps
>>> 1:8, 2:9, 3:10 and so on and assign the values to a new array. For  
>>> this
>>> I am using the following script.
>>> for(i in 1:243)
>>> {
>>> for(j in 1:246)
>>> {
>>> for(k in 1:768)
>>> {
>>> newVar[i,j,k] <- mean( myVar[i,j,k:k+7])
>>> }
>>> }
>>> }
>>> This works, but needless to mention it take a very long time to loop
>>> over all the rows, columns and time periods. I was wondering if  
>>> there is
>>> a simpler way to do this.
>>
>> Yes, vectorize it.  I think this would work:
>>
>> newVar <- array(NA, c(243, 246, 768))
>>
>> for (k in 1:768)
>> newVar[,,k] <- apply(myVar, 1:2, function(x) mean(x[k:(k+7)]))
>
> That's rather interesting. I had not realized that one could use  
> that construction with apply. I had earlier tried to substitute  
> rollmean for the inner loop. For one thing I thought that trying to  
> index 768+7 was going to create some problems with out-of-range  
> indexing. I got the same error with my effort to insert rollmean in  
> the OP's construction as I do in this construction:
>
> myVar <- array(1:(243*246*768), dim=c(243,246,768))
> > myVar[1,1,1:8]
> [1]      1  59779 119557 179335 239113 298891 358669 418447
>
> newVar <-array(,dim=c(243,246,768))
> library(zoo)
> for (k in 1:768)  newVar[,,k] <- apply(myVar, 1:2, function(x)  
> mean(x[k:(k+7)]))
> Error in newVar[, , k] <- apply(myVar, 1:2, function(x) rollmean(x,  
> 8)) :
>  number of items to replace is not a multiple of replacement length
>
> I am guessing that at some point the assignment function cannot  
> resolve which index in newVar to use. So I tried redimensioning  
> newVar to only have 761 as it third dimension and using:
>
> myVar[1,1,1:8]
>
> for (k in 1:761)  {newVar[ , , ] <- apply(myVar, 1:2, function(x)  
> mean(x[k:(k+7)])) ; print(k)}

I forgot to mention that at this point I decided that the for loop  
might be entirely superfluous if one were returning a 761 length  
vector from rollmean. So I tested my theory:

>
> > newVar <-array(,dim=c(243,246,761))
> > str(newVar); Sys.time()
> logi [1:243, 1:246, 1:761] NA NA NA NA NA NA ...
> [1] "2009-02-27 22:35:48 EST"
> >  newVar[,,] <- apply(myVar, 1:2, function(x) rollmean(x, 8));  
> Sys.time()
> [1] "2009-02-27 22:35:57 EST"
> > str(newVar)
> num [1:243, 1:246, 1:761] 209224 269002 328780 388558 448336 ...
> >
>
> -- 
> David Winsemius
> Heritage Laboratories