[R] Filling a covariance matrix

[Joshua Ulrich]

Hi Eric,

On Tue, Oct 23, 2012 at 10:59 AM, emorway <emorway at usgs.gov> wrote:
> useRs –
> I’m working with the attached data that contains one year’s worth of
> sub-daily observations of flow (“Q”) and specific conductance (“SC”, a
> surrogate for concentration) at a point in a stream.  The R code posted
> below shows the extent of data processing thus far.  My goal is to create a
> covariance matrix that takes on the following form:
>       Q1 Q2 … Q365  SC1 SC2 … SC365
> Q1
> Q2
> …
> Q365
> SC1
> SC2
> ...
> SC365
>
> Where the covariance between Q1 (flow on day 1) & Q2 (flow on day 2) is
> determined using the sub-daily data contained in the variable ‘x’ of the R
> code below.  Similarly, the covariance between Q1 & SC1 (specific
> conductance on day 1) would be made using the sub-daily observations of flow
> and specific conductance.  Covariance between observations that are more
> than 5 days distant from one another are likely meaningless.  Thus, the
> covariance matrix should reflect this limitation with zeros.  For example,
> the covariance between Q1 & Q6, or between Q1 & SC6, or between SC359
> (specific conductance on day 359) & SC365 (specific conductance on day 365)
> would be zero as these observations are more than 5 days apart.  Here is the
> R code that reads the attached files containing Q and SC and puts the
> processed data into ‘x’:
>
> 07130500_BelowJM_q_2004.txt
> <http://r.789695.n4.nabble.com/file/n4647170/07130500_BelowJM_q_2004.txt>
> 07130500_BelowJM_2004.txt
> <http://r.789695.n4.nabble.com/file/n4647170/07130500_BelowJM_2004.txt>
>
> library(xts)
> Q_subDaily<-read.table("C:/temp/07130500_BelowJM_q_2004.rdb",col.names=c('date','time','tz','Q','rating','unknown'),colClasses=c("character","character","character","numeric","character","character"))
> SC_subDaily<-read.table("C:/temp/07130500_BelowJM_2004.rdb",col.names=c('date','time','tz','SC','rating','unknown'),colClasses=c("character","character","character","numeric","character","character"))
>
> Q_subDaily$datetime.str <- paste(Q_subDaily$date, Q_subDaily$time)
> SC_subDaily$datetime.str <- paste(SC_subDaily$date, SC_subDaily$time)
>
> fmt <- "%Y%m%d %H%M%S"
> xQ <- xts(as.matrix(Q_subDaily["Q"]), as.POSIXct(Q_subDaily$datetime.str,
> format=fmt))
> xSC <- xts(as.matrix(SC_subDaily["SC"]),
> as.POSIXct(SC_subDaily$datetime.str, format=fmt))
>
> x <- merge(xQ,xSC)
>
> And here’s where I’m stuck, I’m not sure how to create the covariance matrix
> I’ve described above?  I would appreciate and greatly benefit from the sort
> of help often found in the useR community.
>
Thanks for the reproducible example.  I don't have time to provide a
complete solution, but this should get you started and hopefully give
you some ideas:

# create a list of xts objects that only contain data for one day
xs <- split(x, "days")
# initialize the covariance matrix
xcov <- matrix(0, length(xs)*2, length(xs)*2)
rownames(xcov) <- colnames(xcov) <-
  c(sprintf("Q%d",seq_along(xs)), sprintf("SC%d",seq_along(xs)))
# use a double-for loop to fill it in
for(i in seq_along(xs)) {
  for(j in seq_along(xs)) {
    xcov[paste0("Q",i),paste0("Q",j)] <- # same as below
    xcov[paste0("Q",j),paste0("Q",i)] <- cov(xs[[i]]$Q, xs[[j]]$Q, use="pair")
    xcov[paste0("Q",i),paste0("SC",j)] <- # same as below
    xcov[paste0("Q",j),paste0("SC",i)] <- cov(xs[[i]]$Q, xs[[j]]$SC, use="pair")
    xcov[paste0("SC",i),paste0("SC",j)] <- # same as below
    xcov[paste0("SC",j),paste0("SC",i)] <- cov(xs[[i]]$SC, xs[[j]]$SC,
use="pair")
  }
}

>
>
>
> --
> View this message in context: http://r.789695.n4.nabble.com/Filling-a-covariance-matrix-tp4647170.html
> Sent from the R help mailing list archive at Nabble.com.
>
Please include some context if you reply via nabble.  This is a
mailing list, not a forum, and nabble doesn't quote the previous post
by default, which can be frustrating for mailing list subscribers.

> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Best,
--
Joshua Ulrich  |  about.me/joshuaulrich
FOSS Trading  |  www.fosstrading.com