[R] netcdf data precision or least significant digit
Ismail SEZEN
sezenismail at gmail.com
Fri Jul 8 03:36:32 CEST 2016
Thank you very much Roy and Jeff for your help. I contacted dear David, the
author of ncdf4 package, at first and he lead me here. Sorry for any
inconvenience.
I wish you success in your work.
Regards,
On Fri, Jul 8, 2016, 04:28 Roy Mendelssohn - NOAA Federal <
roy.mendelssohn at noaa.gov> wrote:
> I have moved this over to the netcdf-group mail list, which I think is the
> more appropriate place at this point. You are copied, and hopefully
> someone from ESRL will see it and provide the proper response.
>
> HTH,
>
> -Roy
>
> > On Jul 7, 2016, at 6:02 PM, Ismail SEZEN <sezenismail at gmail.com> wrote:
> >
> > Thank you Roy. If I use "round(uwind, digits = 2)”, all data will have 2
> decimal places after decimal point. It’s ok. But How do you know you should
> round the number to 2 decimal digits? According to definitions of precision
> and least_significant_digit, should I round to 2 decimal digits or 1
> decimal digit?
> >
> > For instance, If you check the header information of omega.2015.nc file
> it says;
> >
> > $ ncdump -h omega.2015.nc
> >
> > ...
> > omega:precision = 3s;
> > omega:least_significant_digit = 3s;
> > …
> >
> > So, I need to round values to 3 decimal places after point?
> >
> > and if you check the output of rhum.2015.nc;
> >
> > $ ncdump -h rhum.2015.nc
> > ...
> > rhum:precision = 2s ;
> > rhum:least_significant_digit = 0s ;
> > …
> >
> > Then I need to round values to 2 decimal places after point?
> >
> > Should I accomplish the rounding operation according to precision or
> least_significant_digit attributes? I think someone put these attributes in
> netcdf files for some reason. Also I belive, if required, this kind of an
> operation must be done in related package but author said that it is
> nothing to do with ncdf4 package.
> >
> > Please, forgive me for taking your time.
> >
> >
> >> On 08 Jul 2016, at 03:21, Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn at noaa.gov> wrote:
> >>
> >> After looking at the file, doing an extract say into the variable
> uwind, if I do:
> >>
> >> str(uwind)
> >>
> >> I see what I expect, but if I just do:
> >>
> >> uwind
> >>
> >>
> >> I see what you are seeing. Try:
> >>
> >> uwindnew <- round(uwind, digits = 2)
> >>
> >>
> >> and see if that gives you the results you would expect.
> >>
> >> HTH,
> >>
> >> -Roy
> >>
> >>> On Jul 7, 2016, at 4:49 PM, Ismail SEZEN <sezenismail at gmail.com>
> wrote:
> >>>
> >>> Thank you Roy.
> >>>
> >>> I use NCEP/NCAR Reanalysis 2 data [1]. More precisely, u-wind data of
> the year 2015 [2]. I am also pretty sure that the variables like
> scale_factor or add_offset should be precise like 0.01 or 187.65 but
> somehow (I hope this is not an issue originated by me) they are not,
> including data. Also let me note that I already contacted to author of
> ncdf4 package and also sent an email to ESRL, too, but no luck yet.
> >>>
> >>> For a vectoral data, absolute and mutual u components of wind speed at
> the poles must be equal. For instance, at “2015-01-01 00 GMT”, u-wind at
> longitude=0 and latitude=90 is 9.1999979 m/s and u-wind at longitude=180
> and latitude=90 is -9.2000017 m/s. Minus sign comes from positive north
> direction. Physically, their absolute values must be equal.
> >>>
> >>> 1-
> http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis2.html
> >>> 2-
> ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis2.dailyavgs/pressure/uwnd.2015.nc
> >>>
> >>>
> >>>
> >>>> On 08 Jul 2016, at 02:27, Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn at noaa.gov> wrote:
> >>>>
> >>>> Hi Ismail:
> >>>>
> >>>> Can you point me to a particular netcdf file you are working with. I
> would like to play with it for awhile. I am pretty certain the scale
> factor is 0.01 and what you are seeing in rounding error (or mor precisely
> I should say problems with representations of floating point numbers), but
> i would like to see if there is away around this.
> >>>>
> >>>> Thank,
> >>>>
> >>>> -Roy
> >>>>
> >>>>> On Jul 7, 2016, at 4:16 PM, Ismail SEZEN <sezenismail at gmail.com>
> wrote:
> >>>>>
> >>>>> Thank you very much Jeff. I think I’m too far to be able to explain
> myself. Perhaps, this is the wrong list for this question but I sent it in
> hope there is someone has deep understanding of netcdf data and use R. Let
> me tell the story simpler. Assume that you read a numeric vector of data
> from a netcdf file:
> >>>>>
> >>>>> data <- c(9.1999979, 8.7999979, 7.9999979, 3.0999980, 6.1000018,
> 10.1000017, 10.4000017, 9.2000017)
> >>>>>
> >>>>> you know that the values above are a model output and also you know
> that, physically, first and last values must be equal but somehow they are
> not.
> >>>>>
> >>>>> And now, you want to use “periodic” spline for the values above.
> >>>>>
> >>>>> spline(1:8, data, method = “periodic”)
> >>>>>
> >>>>> Voila! spline method throws a warning message: “spline: first and
> last y values differ - using y[1] for both”. Then I go on digging and
> discover 2 attributes in netcdf file: “precision = 2” and
> “least_significant_digit = 1”. And I also found their definitions at [1].
> >>>>>
> >>>>> precision -- number of places to right of decimal point that are
> significant, based on packing used. Type is short.
> >>>>> least_significant_digit -- power of ten of the smallest decimal
> place in unpacked data that is a reliable value. Type is short.
> >>>>>
> >>>>> Please, do not condemn me, english is not my main language :). At
> this point, as a scientist, what would you do according to explanations
> above? I think I didn’t exactly understand the difference between precision
> and least_significant_digit. One says “significant” and latter says
> “reliable”. Should I round the numbers to 2 decimal places or 1 decimal
> place after decimal point?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> 1-
> http://www.esrl.noaa.gov/psd/data/gridded/conventions/cdc_netcdf_standard.shtml
> >>>>>
> >>>>>
> >>>>>> On 08 Jul 2016, at 01:29, Jeff Newmiller <jdnewmil at dcn.davis.ca.us>
> wrote:
> >>>>>>
> >>>>>> Correction:
> >>>>>>
> >>>>>> ?options (not par)
> >>>>>> --
> >>>>>> Sent from my phone. Please excuse my brevity.
> >>>>>>
> >>>>>> On July 7, 2016 3:26:06 PM PDT, Jeff Newmiller <
> jdnewmil at dcn.davis.ca.us> wrote:
> >>>>>>> Same as with any floating point numeric computation environment...
> you
> >>>>>>> don't. There is always uncertainty in any floating point number...
> it
> >>>>>>> is just larger in this data than you might be used to.
> >>>>>>>
> >>>>>>> Once you get to the stage where you want to output values, read up
> on
> >>>>>>>
> >>>>>>> ?round
> >>>>>>> ?par (digits)
> >>>>>>>
> >>>>>>> and don't worry about the incidental display of extra digits prior
> to
> >>>>>>> presentation (output).
> >>>>>>> --
> >>>>>>> Sent from my phone. Please excuse my brevity.
> >>>>>>>
> >>>>>>> On July 7, 2016 12:50:54 AM PDT, Ismail SEZEN <
> sezenismail at gmail.com>
> >>>>>>> wrote:
> >>>>>>>> Hello,
> >>>>>>>>
> >>>>>>>> I use ncdf4 and ncdf4.helpers packages to get wind data from
> ncep/ncar
> >>>>>>>> reanalysis ncetcdf files. But data is in the form of (9.199998,
> >>>>>>>> 8.799998, 7.999998, 3.099998, -6.8000018, …). I’m aware of
> precision
> >>>>>>>> and least_significant_digit attributes of ncdf4 object [1]. For
> uwnd
> >>>>>>>> data, precision = 2 and least_significant_digits = 1. My doubt is
> that
> >>>>>>>> should I round data to 2 decimal places or 1 decimal place after
> >>>>>>>> decimal point?
> >>>>>>>>
> >>>>>>>> Same issue is valid for some header info.
> >>>>>>>>
> >>>>>>>> Output of ncdf4 object:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Output of ncdump on terminal:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> for instance, ncdump's scale factor is 0.01f but ncdf4 object’s
> >>>>>>>> scale_factor is 0.00999999977648258. You can notice same issue for
> >>>>>>>> actual_range and add_offset. Also a similar issue exist for the
> data.
> >>>>>>>> How can I truncate those extra unsignificant decimal places or
> round
> >>>>>>>> the numbers to significant decimal places?
> >>>>>>>>
> >>>>>>>> 1 -
> >>>>>>>>
> http://www.esrl.noaa.gov/psd/data/gridded/conventions/cdc_netcdf_standard.shtml
> >>>>>>>> <
> http://www.esrl.noaa.gov/psd/data/gridded/conventions/cdc_netcdf_standard.shtml
> >
> >>>>>>>> ______________________________________________
> >>>>>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>>>>> PLEASE do read the posting guide
> >>>>>>>> http://www.R-project.org/posting-guide.html
> >>>>>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>>>>
> >>>>>>> ______________________________________________
> >>>>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>>>> PLEASE do read the posting guide
> >>>>>>> http://www.R-project.org/posting-guide.html
> >>>>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>>>
> >>>>>
> >>>>>
> >>>>> [[alternative HTML version deleted]]
> >>>>>
> >>>>> ______________________________________________
> >>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> >>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>
> >>>> **********************
> >>>> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> >>>> **********************
> >>>> Roy Mendelssohn
> >>>> Supervisory Operations Research Analyst
> >>>> NOAA/NMFS
> >>>> Environmental Research Division
> >>>> Southwest Fisheries Science Center
> >>>> ***Note new address and phone***
> >>>> 110 Shaffer Road
> >>>> Santa Cruz, CA 95060
> >>>> Phone: (831)-420-3666
> >>>> Fax: (831) 420-3980
> >>>> e-mail: Roy.Mendelssohn at noaa.gov www: http://www.pfeg.noaa.gov/
> >>>>
> >>>> "Old age and treachery will overcome youth and skill."
> >>>> "From those who have been given much, much will be expected"
> >>>> "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
> >>>>
> >>>
> >>
> >> **********************
> >> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> >> **********************
> >> Roy Mendelssohn
> >> Supervisory Operations Research Analyst
> >> NOAA/NMFS
> >> Environmental Research Division
> >> Southwest Fisheries Science Center
> >> ***Note new address and phone***
> >> 110 Shaffer Road
> >> Santa Cruz, CA 95060
> >> Phone: (831)-420-3666
> >> Fax: (831) 420-3980
> >> e-mail: Roy.Mendelssohn at noaa.gov www: http://www.pfeg.noaa.gov/
> >>
> >> "Old age and treachery will overcome youth and skill."
> >> "From those who have been given much, much will be expected"
> >> "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
> >>
> >
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new address and phone***
> 110 Shaffer Road
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn at noaa.gov www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
>
[[alternative HTML version deleted]]
More information about the R-help
mailing list