[R] ncdf4: Why are NAs converted to _FillValue when saving?

[David W. Pierce]

On Mon, Aug 14, 2017 at 5:29 AM, <raphael.felber at agroscope.admin.ch> wrote:

Dear all
>
> I'm a newbie regarding netcdf data. Today I realized that I maybe do not
> understand some basics of the netcdf. I want to create a *.nc file
> containing three variables for Switzerland. All data outside of the country
> are NAs. The third variable is calculated from the first two variables.
> Basically there is no problem to do that. I copy the file with the data of
> the first variable, open this file with 'write=TRUE' (nc1 <- nc_open()),
> read the data to 'var1', open the other file (nc2 <- nc_open()), read the
> data to variable 'var2', put this variable to the file (nc1) and calculate
> the third variable based on var1 and var2.
>
> So far everything is fine. But I figured out that when I write the data
> 'var2' to nc1, all NAs in this variable are converted to the
> _FillValue-value. Clearly, I expect that all NAs are converted to the
> _FillValue in the file, but I do not expect that also the NAs in 'var2'
> (i.e. the data which can be called in the R-console) is changed. Since I
> use this data for further calculations, the NAs should remain.
>
> Is that a bug or intended? Below you find a minimal example (adapted from
> the code in the netcdf4 manual) of the – in my eye – strange behavior.
>

​HI Raphael,

I'm going to claim that this is more of an R question than a ncdf4 question
per se. For example, you will notice that if you multiply data_temp2 times
1.0 (leaving values unchanged) or add zero to data_temp2, then the behavior
is what you are expecting. Same holds if you multiply data_temp by 1.0 or
add zero to it.  It would seem that R does the equivalent of assigning
another pointer to the data stored in data_temp rather than copying
data_temp until ​either data_temp or data_temp2 is operated upon, at which
point a copy is made. I personally did not realize this was part of R's
magic.

Regards,

--Dave


>
>
>
> Minimal working example (adapted from netcdf4 manual):
>
> library(ncdf4)
> #----------------
> # Make dimensions
> #----------------
> xvals <- 1:360
> yvals <- -90:90
> nx <- length(xvals)
> ny <- length(yvals)
> xdim <- ncdim_def('Lon','degreesE', xvals )
> ydim <- ncdim_def('Lat','degreesE', yvals )
> tdim <- ncdim_def('Time','days since 1900-01-01', 0, unlim=TRUE )
> #---------
> # Make var
> #---------
> mv <- 1.e30     # missing value
> var_temp <- ncvar_def('Temperature','K', list(xdim,ydim,tdim), mv )
> #---------------------
> # Make new output file
> #---------------------
> output_fname <-'test_real3d.nc'
> ncid_new <- nc_create( output_fname, list(var_temp))
> #-------------------------------
> # Put some test data in the file
> #-------------------------------
> data_temp <- array(0.,dim=c(nx,ny,1))
> for( j in 1:ny )
> for( i in 1:nx )
> data_temp[i,j,1] <- sin(i/10)*sin(j/10)
>
> # add some NAs
> data_temp[1:10, 1:5, 1] <- NA
>
> # copy data
> data_temp2 <- data_temp
>
> # show what we have
> data_temp[1:12, 1:7, 1]
> data_temp2[1:12, 1:7, 1]
>
> # write to netCDF connection
> ncvar_put( ncid_new, var_temp, data_temp, start=c(1,1,1), count=c(nx,ny,1))
>
> # show what we have now
> data_temp[1:12, 1:7, 1]
> data_temp2[1:12, 1:7, 1]
>
> # Why are there no more NAs in data_temp?  ncvar_put changed NAs to
> _FillValue-value
> # But why are the NAs in data_temp2 also changed to _FillValue?
> #--------------------------
> # Close
> #--------------------------
> nc_close( ncid_new )
>
> ------------------------------------------------------------
> ------------------------
> Raphael Felber, Dr. sc.
> Wissenschaftlicher Mitarbeiter, Klima und Lufthygiene
>
> Eidgenössisches Departement für
> Wirtschaft, Bildung und Forschung WBF
> Agroscope
> Forschungsbereich Agrarökologie und Umwelt
>
> Reckenholzstrasse 191, 8046 Zürich
> Tel. 058 468 75 11
> Fax 058 468 72 01
> raphael.felber at agroscope.admin.ch<mailto:raphael.felber at agroscope.admin.ch
> >
> www.agroscope.ch<http://www.agroscope.ch/>
>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posti
> ng-guide.html
> and provide commented, minimal, self-contained, reproducible code.




-- 
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography, La Jolla, California, USA
(858) 534-8276 (voice)  /  (858) 534-8561 (fax)    dpierce at ucsd.edu

	[[alternative HTML version deleted]]