[R] Memory errors using lmer

[Ben Bolker]

McCall, Ken (CMG-Dayton <Ken.McCall <at> coxinc.com> writes:

> I'm trying to run a linear mixed effects analysis on fairly large
> datasets with lmer (from the lme4 package) on a 32-bit Windows
> machine running XP with 3 GB of RAM. It's not working. (details
> below)
 
> I've researched the ff and bigmemory packages, but it appears they
> won't handle the mixed mode dataset I'm analyzing. It has some
> character fields for the categorical variables. It's also not clear
> a linear mixed effect regression can be run with those packages. Can
> anyone point me to a lme solution on larger datasets that can
> address the 32-bit memory limitations?

 
> The smallest dataframe I'm trying to run is about 120,000
> observations and 7 variables, but I'd rather run a for loop script
> on 620K+ observations. I've seen several recommendations (Kabacoff
> in 'R in Action' and others) that when possible, run R in a 64-bit
> build. Problem is I'm on a deadline, and procuring a new computer
> takes time, and approvals up the food chain. Suggestions? Is 64-bit
> my only option?

> > install.packages("lme4")

  (only needs to be done once)

> > library(lme4)
> > math07g4 <- sqlQuery(conn, "select ssid, ss_chg, 
> campus2, district_id, pblack, pfreelnch, pmob
>          FROM codemob0607ma WHERE grade2 = 4")

  Why are you attach()ing?  Probably unnecessary ...

> > attach(math07g4)


> > fit07ma4 <- lmer(ss_chg ~ 1 + factor(campus2) + factor(district_id) + 
>  pblack + pfreelnch + pmob +
>    (1 | campus2) + (1 | district_id), data=math07g4)
> 
> And I get this:

> Error: cannot allocate vector of size 2.5 Gb
> In addition: Warning messages:
> 1: In model.matrix.default(mt, mf, contrasts) :
>   Reached total allocation of 2187Mb: see help(memory.size)

  Once upon a time there may have been an option for sparse
model matrices, but not now (I think).

  Depending on whether you have any budget at all,
I wonder if you could use Amazon ... google "r amazon ec2 instance"
for more information ...

  If you need more info, I would suggest posting to 
r-sig-mixed-models <at> r-project.org (a specialty mailing 
list for mixed models).