[R] Extracting matrix from netCDF file using ncdf4 package

[Roy Mendelssohn - NOAA Federal]

Sending this to Hemant a second time as i forgot to reply to list.

Hi Hemant:

Well technically the code you give below shouldn’t work, because “start” and “count” are suppose to be of the same dimensions as the variables.  I guess Pierce’s code must be very forgiving if that is working.  One thing you can do to speed things up is pre-allocate the array you want to create, say

> dX <- array(NA_real_, dim=c(5,365))


and then have the ncvar_get call write directly to the array:

> dX[i,] <- ncvar_get(nc=myNC, varid="myVar", start=c(reqX[i],reqY[i],1), count=c(1,1,-1)) 

The second thing you can do, is to use “lapply” instead of the “for” loop, but I don’t know how much faster that will make your code.  The fastest however, if you have the memory, is to just read the array into memory:

> dX <-  ncvar_get(nc=myNC, varid=“myVar”)


and then use R’s subsetting abilities. You can do fancier subsetting of arrays in memory than you can to arrays on disk.

HTH,

-Roy


> On Jul 2, 2016, at 3:43 PM, Hemant Chowdhary via R-help <r-help at r-project.org> wrote:
> 
> I am working with a 3-dimensional netCDF file having dimensions of X=100, Y=200, T=365. 
> My objective is to extract time vectors of a few specific grids that may not be contiguous on X and/or Y. 
> 
> For example, I want to extract a 5x365 matrix where 5 rows are each vectors of length 365 of 5 specific X,Y combinations. 
> 
> For this, I am currently using the following 
> 
> reqX = c(35,35,40,65,95); 
> reqY = c(2,5,10,112,120,120); 
> nD = length(reqX) 
> for(i in 1:nD){ 
> idX = ncvar_get(nc=myNC, varid="myVar", start=c(reqX[i],reqY[i]), count=c(1,1)) 
> if(i==1){dX = idX} else {dX = rbind(dX,idX)} 
> } 
> 
> Is there more elegant/faster way other than to using a For Loop like this? It seems very slow when I may have to get much larger matrix where nD can be more than 1000. 
> 
> Thank you HC
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

> On Jul 2, 2016, at 3:43 PM, Hemant Chowdhary via R-help <r-help at r-project.org> wrote:
> 
> I am working with a 3-dimensional netCDF file having dimensions of X=100, Y=200, T=365. 
> My objective is to extract time vectors of a few specific grids that may not be contiguous on X and/or Y. 
> 
> For example, I want to extract a 5x365 matrix where 5 rows are each vectors of length 365 of 5 specific X,Y combinations. 
> 
> For this, I am currently using the following 
> 
> reqX = c(35,35,40,65,95); 
> reqY = c(2,5,10,112,120,120); 
> nD = length(reqX) 
> for(i in 1:nD){ 
> idX = ncvar_get(nc=myNC, varid="myVar", start=c(reqX[i],reqY[i]), count=c(1,1)) 
> if(i==1){dX = idX} else {dX = rbind(dX,idX)} 
> } 
> 
> Is there more elegant/faster way other than to using a For Loop like this? It seems very slow when I may have to get much larger matrix where nD can be more than 1000. 
> 
> Thank you HC
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn at noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.