# [R] Reshaping an array - how does it work in R

Roy Mendelssohn - NOAA Federal roy.mendelssohn at noaa.gov
Sat Mar 19 17:03:00 CET 2016

```> On Mar 19, 2016, at 8:18 AM, Henrik Bengtsson <henrik.bengtsson at gmail.com> wrote:
>
> On Fri, Mar 18, 2016 at 8:28 PM, Roy Mendelssohn - NOAA Federal
> <roy.mendelssohn at noaa.gov> wrote:
>> Hi Henrik:
>>
>> I want to do want in oceanography is called an EOF, which is just a PCA analysis. Unless I am missing something, in R I need to flatten my 3-D matrix into a 2-D data matrix. I can fit the entire 30GB matrix into memory, and I believe I have enough memory to do the PCA by constraining the number of components returned .  What I don’t think I have enough memory for is an operation that makes a copy of the matrix.
>>
>> As I said, in theory I know how to do the flattening, it a simple command, but in practice I don’t have enough memory.  So I spent the afternoon rewriting my code to read in parts of the data at a time and then putting those in the appropriate places of a matrix already flattened of appropriate size.  In case someone is wondering, on the 3D grid the matrix is [1001,1001,3650].  So I create an empty matrix size [1001*1001,3650], and read in a slice of the lat-lon grid, and map those into the appropriate places in the flattened matrix.  By reading in appropriately sized chunks  my memory usage is not pushed too far.
>
> Sounds good.  There's another small caveat. Make sure to specify the
> 'data' argument for matrix() we allocating an "empty" matrix, e.g.
>
>    X <- matrix(NA_real_, nrow=1001*1001, ncol=3650)
>
> This will give you a double matrix with all missing value.  If you use
> the default
>
>    X <- matrix(nrow=1001*1001, ncol=3650)
>
> you'll get a logical matrix, which will introduce a copy as soon as
> you assign a double value (e.g. X[1,1] <- 3.14). The latter is a
> complete waste of memory/time. See
> http://www.jottr.org/2014/06/matrixNA-wrong-way.html for details.
>
> /Henrik

Thanks.  Yes one time for some reason I can’t remember I did ?NA  where that is documented but it is not something you would think of offhand.

-Roy

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center