[R] problem with loop to put data into array with missing data forsome files

Jenny Barnes jmb at mssl.ucl.ac.uk
Mon Nov 20 15:18:19 CET 2006


Hi John,

I just realised that the file name I gave on the original message was wrong - it 
was Feb 1974 (which is file /home/jmb/sst_precip/gribdata/197402.grib).

If you find that you can download that fine then I would appreciate a copy! 
However you are right, I am after a general solution to help me in the future.

Thank you for the advice on my test for na, I had seen is.na but I was not sure 
if I could use it in this case - I will give it a try!

Thanks again,

Jenny

>
>
>Hi Jenny
>
>If you want a general solution I understand. 
>
>However I just downloaded the file fine (as far as I can tell) so you
>are welcome to have a copy. I can email it to you if you want.
>
>I do not think your test for NA is valid. i.e
>
>if(test != "NA"){
>
>}
>
>I think you should use 
>
>if(is.na(test)){
>
>}
>
>Or something similar.
>
>
>
>J
>
>
> 
>---
>
>John Seers
>Institute of Food Research
>Norwich Research Park
>Colney
>Norwich
>NR4 7UA
> 
>
>tel +44 (0)1603 251497
>fax +44 (0)1603 507723
>e-mail john.seers at bbsrc.ac.uk                         
>e-disclaimer at http://www.ifr.ac.uk/edisclaimer/ 
> 
>Web sites:
>
>www.ifr.ac.uk   
>www.foodandhealthnetwork.com
>
>
>-----Original Message-----
>From: Jenny Barnes [mailto:jmb at mssl.ucl.ac.uk] 
>Sent: 20 November 2006 13:52
>To: john seers (IFR)
>Subject: RE: [R] problem with loop to put data into array with missing
>data forsome files
>
>
>John,
>
>Yes I did try that too, should have mentioned that I guess! It just
>stops 
>downloading half way through - I would like to figure out how to deal
>with 
>incomplete data files incase this happens again when the data is simply
>not 
>available, not just inaccessable!
>
>Thank you very much for your time, if you have any other ideas I would
>love to 
>hear them!
>
>Jenny
>
>>X-MimeOLE: Produced By Microsoft Exchange V6.5
>>Content-class: urn:content-classes:message
>>MIME-Version: 1.0
>>Subject: RE: [R] problem with loop to put data into array with missing
>data 
>forsome files
>>Date: Mon, 20 Nov 2006 13:49:23 -0000
>>X-MS-Has-Attach: 
>>X-MS-TNEF-Correlator: 
>>Thread-Topic: [R] problem with loop to put data into array with missing
>data 
>forsome files
>>Thread-Index: AccMpQnCuMyNH96jTXWVS24vVMWzDgABVhVQ
>>From: "john seers \(IFR\)" <john.seers at bbsrc.ac.uk>
>>To: "Jenny Barnes" <jmb at mssl.ucl.ac.uk>, <r-help at stat.math.ethz.ch>
>>X-OriginalArrivalTime: 20 Nov 2006 13:49:23.0562 (UTC) 
>FILETIME=[AFBB74A0:01C70CAA]
>>X-ECS-MailScanner-BBSRC: Found to be clean
>>X-MSSL-MailScanner-Information: Please contact the ISP for more
>information
>>X-MSSL-MailScanner: No virus found
>>X-MSSL-MailScanner-SpamCheck: not spam, SpamAssassin (score=-4.9,
>required 5, 
>BAYES_00 -4.90)
>>Content-Transfer-Encoding: 8bit
>>X-MIME-Autoconverted: from quoted-printable to 8bit by
>msslsc.mssl.ucl.ac.uk id 
>kAKDoe711827
>>
>>
>>
>>
>>Hi Jenny
>>
>>I guess this is a silly question so sorry in advance. But why don't you
>>just re-download the file that failed to download completely?
>>
>>John
>>
>>
>>
>>
>>
>>
>> 
>>---
>>
>>John Seers
>>Institute of Food Research
>>Norwich Research Park
>>Colney
>>Norwich
>>NR4 7UA
>> 
>>
>>tel +44 (0)1603 251497
>>fax +44 (0)1603 507723
>>e-mail john.seers at bbsrc.ac.uk                         
>>e-disclaimer at http://www.ifr.ac.uk/edisclaimer/ 
>> 
>>Web sites:
>>
>>www.ifr.ac.uk   
>>www.foodandhealthnetwork.com
>>
>>
>>-----Original Message-----
>>From: r-help-bounces at stat.math.ethz.ch
>>[mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Jenny Barnes
>>Sent: 20 November 2006 13:03
>>To: r-help at stat.math.ethz.ch
>>Subject: [R] problem with loop to put data into array with missing data
>>forsome files
>>
>>
>>Dear R-help community,
>>
>>My main goal of this message is to find a way of skipping a file of a
>>month/year 
>>in a loop that does not exist (and making it's output into an data.out
>>array 
>>would be NA) and moving onto the next year/month in the loop to carry
>on
>>filling 
>>data.out with real precipitation data. 
>>
>>The situation so far:
>>I downloaded 50 years worth of GRIB data files from the NCEP data site
>>http://nomad3.ncep.noaa.gov/pub/reanalysis-1/month/grb2d.gau/
>>
>>I then created a loop in R to read each month of each of the 50 years
>>worth of 
>>files and only extract the precipitation records using wgrib and grep
>as
>>show in 
>>the code at the end of this message. I had to use grep to extract the
>>PRATE 
>>records each time, as unfortunately with these grib files precipitation
>>is not 
>>always the same record number each time. 
>>
>>This worked fine until Feb 1972 (which is file 
>>/home/jmb/sst_precip/gribdata/197202.grib). This file did not download 
>>completely and as a result the file that saved to my computer did not
>>include 
>>the record for precipitation - only the first 13 records. At this point
>>in the 
>>loop it just stopped with error message:
>>Error in seq.default(lat.beg, lat.end, length = ny) : 
>>        length must be non-negative number
>>Which is obviously because ny does not actually exist for this file as
>>it cannot 
>>be extracted from source as this record does not exist for this file!
>>
>>My question is:
>>Does anybody have any suggestions as to how I could code it so that my
>>loop will 
>>allow this file to just output NA into the data.out array in
>>data.out$data and 
>>maybe keep FALSE in data.out$date for the month&years where this has
>>occured and 
>>then continue to loop around all the other months and years in
>>succession? 
>>
>>Things that don't work that I have tried:
>>1)I cannot just restart the loop after any anomalous years as this
>>"resets" 
>>data.out and changes its dimensions depending on the number of years
>>looped 
>>(nyrs) etc.
>>2)The first point of trouble is when I call upon ny in the line
>>>lats <- seq(lat.beg, lat.end, length=ny)
>>as ny is NA in this record - so I tried putting the line
>>>test <- dims[15]
>>>if(test != "NA"){
>>with the closing } at the end of the rest of the loop
>>...this gives the error message
>>>Error in if (test != "NA") { : missing value where TRUE/FALSE needed
>>3) My thought was that maybe I would need something like
>>>if (parameter == "PRATE" ){
>>then continue with loop; but this will obviously not work by itself and
>>gives 
>>the error message
>>>Error in if (test != "NA") { : missing value where TRUE/FALSE needed
>>- would need to specify what to do if parameter is not =="PRATE". I
>>encourage 
>>anybody with another idea to contact me with their suggestions!
>>
>>Thank you for taking the time to read my problem,
>>
>>Sincerely,
>>
>>Jennifer Barnes
>>
>>r-help at stat.math.ethz.ch
>>Here is my code:
>>
>>#takes precip data for every year and month needed
>>#from file saved on computer already
>>#and sorts it into a list (array)
>>
>>#set some control parameters
>>
>>startyr <-1954 #note needs to be year before data wanted
>>nyrs <- 50
>>months <- nyrs*12
>>k <- 0
>>library(chron)
>>library(utils)
>>data.out <- list(lats=seq(88.542, -88.542, length=94),
>>		  lons=seq(0, 360-1.875, length=192),
>>		  date=vector(length=months),
>>		  data=array(NA, c(months, 94*192))
>>) 
>>#the loop
>>for(yr in 1:nyrs){
>>     for(mon in 1:12){
>>     	i<- startyr+yr
>>     
>>	if(mon < 10) {j <- 0}
>>	if(mon < 10) {mon2 <- mon}
>>	if(mon == 10) {(j <- 1)}
>>	if(mon == 10) {mon2 <- 0}
>> 	if(mon == 11) {(j <- 1)}
>>	if(mon == 11) {mon2 <- 1}
>>	if(mon == 12) {(j <- 1)}
>>	if(mon == 12) {mon2 <- 2}
>>	
>>	outfile <- paste("/home/jmb/sst_precip/gribdata/",i, j, mon2,
>>".grib", 
>>sep="")  
>>		#decode the grib 
>>	cmd <- paste("/usr/local/bin/wgrib ", outfile, "| grep
>>\":PRATE:\" | 
>>/usr/local/bin/wgrib", outfile, "-i -V -text -o /tmp/wgrib.output")
>>	status <- system(cmd, intern=T)
>>	
>>	#extract metadata needed for working with julian dates:
>>	dt <- unlist(strsplit(status[1], " "))[3]
>>	dt.yr <- as.numeric(substr(dt, 1,4))
>>	dt.mon <- as.numeric(substr(dt, 5,6))
>>	dt.day <- as.numeric(substr(dt, 7,8))
>>	dt.hr <- as.numeric(substr(dt, 9,10))
>>	dt.jul <- julian(dt.mon, dt.day+(dt.hr/24), dt.yr)
>>	
>>	parameter <- unlist(strsplit(status[1], " "))[4]
>>	
>>	param <- status[2]
>>	dims <- unlist(strsplit(status[3], " "))
>>	nx <- as.numeric(dims[13])
>>	ny <- as.numeric(dims[15])
>>	temp <- unlist(strsplit(status[5]," "))
>>	lat.beg <- as.numeric(temp[6])
>>	lat.end <- as.numeric(temp[8])
>>	lats <- seq(lat.beg, lat.end, length=ny)
>>	temp <- unlist(strsplit(status[6]," "))
>>	lon.beg <- as.numeric(temp[14])
>>	lon.end <- as.numeric(temp[16])
>>	if(lon.beg > lon.end){lon.beg <- (lon.beg -360)} #since sp
>>prefers -180 
>>to 180 encoding 
>>	lons <- seq(lon.beg, lon.end, length=nx)
>>	lons[lons > 180] <- lons[lons > 180]-360	 #since sp
>>prefers -180 
>>to 180 encoding 
>>	
>>	#read flat file into R (header first)
>>	dims <- scan("/tmp/wgrib.output", nlines=1, quiet=T)
>>	nrec <- dims[1]*dims[2]
>>	indata <- paste("indata.", i, j, mon2, sep="")
>>	indata <- scan("/tmp/wgrib.output", 'numeric', nlines=nrec,
>>skip=1, 
>>quiet=T)
>>	indata <- matrix(as.numeric(indata), nrow=dims[1], ncol=dims[2],
>>
>>byrow=F)
>>	
>>	#to get precip into mm/month, not mm/sec:
>>	if((dt.mon==02)&(dt.yrendnumber%%4==0)){days.mon <- 29} else
>>{days.mon 
>><-28}    #to take into account leap-years!!!
>>	if(dt.mon==01){days.mon <- 31}
>>	if(dt.mon==03){days.mon <- 31}
>>	if(dt.mon==05){days.mon <- 31}
>>	if(dt.mon==07){days.mon <- 31}
>>	if(dt.mon==08){days.mon <- 31}
>>	if(dt.mon==10){days.mon <- 31}
>>	if(dt.mon==12){days.mon <- 31}
>>	if(dt.mon==04){days.mon <- 30}
>>	if(dt.mon==06){days.mon <- 30}
>>	if(dt.mon==09){days.mon <- 30}
>>	if(dt.mon==11){days.mon <- 30}
>>	sec.mon <- days.mon*24*60*60
>>
>>	newindata <- paste("newindata.",i, j, mon2, sep="")
>>	newindata <- sec.mon*indata
>>	
>>	k<-k+1
>>	data.out$date[k] <- paste(i, j, mon2, sep="")
>>	data.out$data[k,] <- newindata
>>}
>>}
>>
>>
>>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>Jennifer Barnes
>>PhD student - long range drought prediction
>>Climate Extremes
>>Department of Space and Climate Physics
>>University College London
>>Holmbury St Mary, Dorking
>>Surrey
>>RH5 6NT
>>Web: http://climate.mssl.ucl.ac.uk
>>
>>______________________________________________
>>R-help at stat.math.ethz.ch mailing list
>>https://stat.ethz.ch/mailman/listinfo/r-help
>>PLEASE do read the posting guide
>>http://www.R-project.org/posting-guide.html
>>and provide commented, minimal, self-contained, reproducible code.
>
>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>Jennifer Barnes
>PhD student - long range drought prediction
>Climate Extremes
>Department of Space and Climate Physics
>University College London
>Holmbury St Mary, Dorking
>Surrey
>RH5 6NT
>01483 204149
>07916 139187
>Web: http://climate.mssl.ucl.ac.uk
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jennifer Barnes
PhD student - long range drought prediction
Climate Extremes
Department of Space and Climate Physics
University College London
Holmbury St Mary, Dorking
Surrey
RH5 6NT
01483 204149
07916 139187
Web: http://climate.mssl.ucl.ac.uk



More information about the R-help mailing list