[R] Fwd: Variance Components in R
Spencer Graves
spencer.graves at pdf.com
Thu Aug 17 19:46:45 CEST 2006
Burt Gunter just reminded me that the completion time could also
be affected by the numbers of levels of each of the factors, especially
random effects: With N records, any variance components / mixed model
software using MLE or REML will have to invert repeatedly an N x N
matrix for the covariance structure of the random effects and noise. If
the software recognizes your design as having some simple structure,
this can be quite fast; otherwise, it could be a Herculean task. In
your case with N = 9500 records, just one copy of this covariance matrix
could consume a substantial portion of 1GB RAM. I compute
8*9500*(9500-1)/2 = 361Mbytes.
However, any software that recognizes special structure in your
design may be able to do the required computations without ever
constructing a matrix this large. I would say that it's still worth a
try in R on your laptop or on the machine with 1GB RAM: 'lmer' might
recognize special structure that neither of the other two do (and vice
versa).
Hope this helps.
Spencer Graves
Iuri Gavronski wrote:
> We have tried on many machines, from my laptop to a dual core Intel
> processor with 1GB of RAM.
>
> On 8/17/06, *Spencer Graves* < spencer.graves at pdf.com
> <mailto:spencer.graves at pdf.com>> wrote:
>
> Hi, Iuri:
>
> How much RAM and how fast a microprocessor (and what version of
> Windows) do you have? You might still try it in R under Windows. The
> results might be comparable or dramatically better in R than in
> SPSS or
> SAS.
>
> hope this helps.
> Spencer Graves
>
> Iuri Gavronski wrote:
> > 9500 records. It didn`t run in SPSS or SAS on Windows machines,
> so I am
> > trying to convert the SPSS script to R to run in a RISC station
> at the
> > university.
> >
> > On 8/17/06, Doran, Harold < HDoran at air.org
> <mailto:HDoran at air.org>> wrote:
> >
> >> Iuri:
> >>
> >> The lmer function is optimal for large data with crossed random
> effects.
> >> How large are your data?
> >>
> >>
> >>> -----Original Message-----
> >>> From: r-help-bounces at stat.math.ethz.ch
> <mailto:r-help-bounces at stat.math.ethz.ch>
> >>> [mailto: r-help-bounces at stat.math.ethz.ch
> <mailto:r-help-bounces at stat.math.ethz.ch>] On Behalf Of Iuri Gavronski
> >>> Sent: Thursday, August 17, 2006 11:08 AM
> >>> To: Spencer Graves
> >>> Cc: r-help at stat.math.ethz.ch <mailto:r-help at stat.math.ethz.ch>
> >>> Subject: Re: [R] Variance Components in R
> >>>
> >>> Thank you for your reply.
> >>> VARCOMP is available at SPSS advanced models, I'm not sure
> >>> for how long it exists... I only work with SPSS for the last
> >>> 4 years...
> >>> My model only has crossed random effects, what perhaps would
> >>> drive me to lmer().
> >>> However, as I have unbalanced data (why it is normally called
> >>> 'unbalanced design'? the data was not intended to be
> >>> unbalanced, only I could not get responses for all cells...),
> >>> I'm afraid that REML would take too much CPU, memory and time
> >>> to execute, and MINQUE would be faster and provide similar
> >>> variance estimates (please, correct me if I'm wrong on that
> point).
> >>> I only found MINQUE on the maanova package, but as my study
> >>> is very far from genetics, I'm not sure I can use this package.
> >>> Any comment would be appreciated.
> >>> Iuri
> >>>
> >>> On 8/16/06, Spencer Graves < spencer.graves at pdf.com
> <mailto:spencer.graves at pdf.com>> wrote:
> >>>
> >>>> I used SPSS over 25 years ago, but I don't recall
> >>>>
> >>> ever fitting a
> >>>
> >>>> variance components model with it. Are all your random
> >>>>
> >>> effects nested?
> >>>
> >>>> If they were, I would recommend you use 'lme' in the 'nlme'
> package.
> >>>> However, if you have crossed random effects, I suggest you
> >>>>
> >>> try 'lmer'
> >>>
> >>>> associated with the 'lme4' package.
> >>>>
> >>>> For 'lmer', documentation is available in Douglas
> >>>>
> >>> Bates. Fitting
> >>>
> >>>> linear mixed models in R. /R News/, 5(1):27-30, May 2005
> >>>> (www.r-project.org <http://www.r-project.org> ->
> newsletter). I also recommend you try the
> >>>> vignette available with the 'mlmRev' package (see, e.g.,
> >>>> http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html
> <http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html>).
> >>>>
> >>>> Excellent documentation for both 'lme' (and indirectly for
> >>>> 'lmer') is available in Pinheiro and Bates (2000)
> >>>>
> >>> Mixed-Effects Models
> >>>
> >>>> in S and S-Plus (Springer). I have personally recommended
> >>>>
> >>> this book
> >>>
> >>>> so many times on this listserve that I just now got 234 hits for
> >>>> RSiteSearch("graves pinheiro"). Please don't hesitate to
> pass this
> >>>> recommendation to your university library. This book is
> >>>>
> >>> the primary
> >>>
> >>>> documentation for the 'nlme' package, which is part of the
> >>>>
> >>> standard R
> >>>
> >>>> distribution. A subdirectory "~library\nlme\scripts" of your R
> >>>> installation includes files named "ch01.R", "ch02.R", ...,
> >>>>
> >>> "ch06.R",
> >>>
> >>>> "ch08.R", containing the R scripts described in the
> book. These R
> >>>> script files make it much easier and more enjoyable to study that
> >>>> book, because they make it much easier to try the commands
> >>>>
> >>> described
> >>>
> >>>> in the book, one line at a time, testing modifications to
> check you
> >>>> comprehension, etc. In addition to avoiding problems with
> >>>> typographical errors, it also automatically overcomes a few
> >>>>
> >>> minor but
> >>>
> >>>> substantive changes in the notation between S-Plus and R.
> >>>>
> >>>> Also, the "MINQUE" method has been obsolete for over
> >>>>
> >>> 25 years.
> >>>
> >>>> I recommend you use method = "REML" except for when you want to
> >>>> compare two nested models with different fixed effects; in
> >>>>
> >>> that case,
> >>>
> >>>> you should use method = "ML", as explained in Pinheiro and
> >>>>
> >>> Bates (2000).
> >>>
> >>>> Hope this helps.
> >>>> Spencer Graves
> >>>>
> >>>> Iuri Gavronski wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I'm trying to fit a model using variance components in R,
> but if
> >>>>> very new on it, so I'm asking for your help.
> >>>>>
> >>>>> I have imported the SPSS database onto R, but I don't know
> how to
> >>>>> convert the commands... the SPSS commands I'm trying to
> >>>>>
> >>> convert are:
> >>>
> >>>>> VARCOMP
> >>>>> RATING BY CHAIN SECTOR RESP ASPECT ITEM
> >>>>> /RANDOM = CHAIN SECTOR RESP ASPECT ITEM
> >>>>> /METHOD = MINQUE (1)
> >>>>> /DESIGN = CHAIN SECTOR RESP ASPECT ITEM
> >>>>> SECTOR*RESP SECTOR*ASPECT SECTOR*ITEM CHAIN*RESP
> >>>>> CHAIN*ASPECT CHAIN*ITEM RESP*ASPECT RESP*ITEM
> >>>>> SECTOR*RESP*ASPECT SECTOR*RESP*ITEM
> >>>>>
> >>> CHAIN*RESP*ASPECT
> >>>
> >>>>> /INTERCEPT = INCLUDE.
> >>>>>
> >>>>> VARCOMP
> >>>>> RATING BY CHAIN SECTOR RESP ASPECT ITEM
> >>>>> /RANDOM = CHAIN SECTOR RESP ASPECT ITEM
> >>>>> /METHOD = REML
> >>>>> /DESIGN = CHAIN SECTOR RESP ASPECT ITEM
> >>>>> SECTOR*RESP SECTOR*ASPECT SECTOR*ITEM CHAIN*RESP
> >>>>> CHAIN*ASPECT CHAIN*ITEM RESP*ASPECT RESP*ITEM
> >>>>> SECTOR*RESP*ASPECT SECTOR*RESP*ITEM
> >>>>>
> >>> CHAIN*RESP*ASPECT
> >>>
> >>>>> /INTERCEPT = INCLUDE.
> >>>>>
> >>>>> Thank you for your help.
> >>>>>
> >>>>> Best regards,
> >>>>>
> >>>>> Iuri.
> >>>>>
> >>>>> _______________________________________
> >>>>> Iuri Gavronski - iuri at ufrgs.br <mailto:iuri at ufrgs.br>
> >>>>> doutorando
> >>>>> UFRGS/PPGA/NITEC - www.ppga.ufrgs.br
> <http://www.ppga.ufrgs.br> Brazil
> >>>>>
> >>>>> ______________________________________________
> >>>>> R-help at stat.math.ethz.ch <mailto:R-help at stat.math.ethz.ch>
> mailing list
> >>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> <https://stat.ethz.ch/mailman/listinfo/r-help>
> >>>>> PLEASE do read the posting guide
> >>>>>
> >>>> http://www.R-project.org/posting-guide.html
> >>>>
> >>>>> and provide commented, minimal, self-contained, reproducible
> code.
> >>>>>
> >>>>>
> >>> [[alternative HTML version deleted]]
> >>>
> >>> ______________________________________________
> >>> R-help at stat.math.ethz.ch <mailto:R-help at stat.math.ethz.ch>
> mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>> PLEASE do read the posting guide
> >>> http://www.R-project.org/posting-guide.html
> >>> and provide commented, minimal, self-contained, reproducible
> code.
> >>>
> >>>
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at stat.math.ethz.ch <mailto:R-help at stat.math.ethz.ch>
> mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>
More information about the R-help
mailing list