[R] lack of memory for logistic regression in R?

Gavin Simpson gavin.simpson at ucl.ac.uk
Mon Jun 15 18:52:39 CEST 2009


On Mon, 2009-06-15 at 07:35 -0700, Michael wrote:
> Yet what's the benefit of using Design Package?

Given that glm apparently can't operate within the memory limits of your
PC setup on your particular data set/model, perhaps the question should
"what do you have to loose in trying Design?"

If lrm() also won't work, you might try glm.fit supplied with R and used
as the work horse function behind glm. The reason I mention this is that
the problem you might be seeing could be coming from the part of glm
that builds the model matrices and the like rather than from the fitting
part of glm.fit.

You need to provide the design matrix yourself of course.

60000 x 20 doesn't sound like a such a large design matrix - perhaps
there is something you are not telling us about the variables in C? Are
some/all of the 19 predictors factors, with lots of levels? Maybe they
shouldn't be but have ended up that way after being read in to R?

You could do a traceback() straight after the error to see where the
failure was, and maybe also debug glm

debug(glm)

to step through the code as you run your model to see where the code is
bailing out. That will guide you to whether the failure is in the
fitting part or the bit before glm.fit is called which is where the
model (design) matrices are computed etc.

Another alternative is to try the biglm package which can fit GLMs on
huge data sets.

HTH

G

> 
> Thanks!
> 
> On Sun, Jun 14, 2009 at 8:04 PM, Frank E Harrell
> Jr<f.harrell at vanderbilt.edu> wrote:
> > Also it would be useful to compare glm with the lrm function in the Design
> > package, for speed and memory use.
> >
> > Frank
> >
> >
> > David Winsemius wrote:
> >>
> >> On Jun 14, 2009, at 9:06 PM, Michael wrote:
> >>
> >>> Hi all,
> >>>
> >>> I am getting the following error message:
> >>>
> >>>> mymodel = glm(response ~ . , family=binomial, data=C);
> >>>
> >>> Error: cannot allocate vector of size 734.2 Mb
> >>> In addition: Warning messages:
> >>> 1: In array(0, c(n, n), list(levs, levs)) :
> >>>  Reached total allocation of 1535Mb: see help(memory.size)
> >>> 2: In array(0, c(n, n), list(levs, levs)) :
> >>>  Reached total allocation of 1535Mb: see help(memory.size)
> >>> 3: In array(0, c(n, n), list(levs, levs)) :
> >>>  Reached total allocation of 1535Mb: see help(memory.size)
> >>> 4: In array(0, c(n, n), list(levs, levs)) :
> >>>  Reached total allocation of 1535Mb: see help(memory.size)
> >>>
> >>> -----------
> >>>
> >>> The data frame is 60000 x 20,
> >>>
> >>> is it too large for R?
> >>>
> >>> What shall I do? Will close all other softwares/applications help? My
> >>> PC is Vista with 4GB memory. Thank you.
> >>
> >> It's certainly not too large for R. Have you looked at the R Windows FAQ
> >> on the topic?
> >>
> >>
> >> http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-be-a-limit-on-the-memory-it-uses_0021
> >>
> >> ... and perhaps:
> >>
> >> http://finzi.psych.upenn.edu/Rhelp08/2008-August/171649.html
> >>
> >>
> >> David Winsemius, MD
> >> Heritage Laboratories
> >> West Hartford, CT
> >>
> >> ______________________________________________
> >> R-help at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide
> >> http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained, reproducible code.
> >>
> >
> >
> > --
> > Frank E Harrell Jr   Professor and Chair           School of Medicine
> >                     Department of Biostatistics   Vanderbilt University
> >
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
-- 
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
 Dr. Gavin Simpson             [t] +44 (0)20 7679 0522
 ECRC, UCL Geography,          [f] +44 (0)20 7679 0565
 Pearson Building,             [e] gavin.simpsonATNOSPAMucl.ac.uk
 Gower Street, London          [w] http://www.ucl.ac.uk/~ucfagls/
 UK. WC1E 6BT.                 [w] http://www.freshwaters.org.uk
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%




More information about the R-help mailing list