[R] bam (mgcv) not using the specified number of cores
andrewcd at gmail.com
Fri Aug 22 07:13:43 CEST 2014
(resending with all images as imgur so as to not bounce from list)
Thanks for the reply. I've tried to reproduce the error, but I don't
know how to show output from `top` any other way than with screenshots,
so please excuse that.
Here are screenshots of what happens when I run with two
cores. In the former, it seems to be working on one core, and in the
latter, it appears to be working on three. When reproducing the error,
I'm getting behavior that isn't entirely consistent -- sometimes it
"behaves" and operates on the asked-for number of cores, and other times
I'm also attaching a screenshot
showing terminal output from a remote cluster when I run my full model
(N=67K) rather than a subset (N=7K) -- I get that error "Error in
qr.qty(qrx, f) : right-hand side should have 60650 not 118451 rows ..."
I suppose this is a memory overload problem? Any suggestions on how to
get bam to not call for more memory than the node has available would be
welcome, though I suspect that is a supercomputing problem rather than a
mgcv problem. I don't know much about memory management, except that R
doesn't do it explicitly.
sessionInfo() for local machine:
R version 3.0.2 (2013-09-25)
Platform: x86_64-pc-linux-gnu (64-bit)
 LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
 LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
 LC_PAPER=en_US.UTF-8 LC_NAME=C
 LC_ADDRESS=C LC_TELEPHONE=C
 LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
 parallel stats graphics grDevices utils datasets methods
other attached packages:
 mgcv_1.7-26 nlme_3.1-111
loaded via a namespace (and not attached):
 grid_3.0.2 lattice_0.20-23 Matrix_1.1-4
On 08/21/2014 04:29 PM, Simon Wood wrote:
> Hi Andrew,
> Could you provide a bit more information, please. In particular the
> results of sessionInfo() and the code that caused this weird behaviour
> (+ an indication of dataset size).
> On 21/08/14 12:53, Andrew Crane-Droesch wrote:
>> I am getting strange behavior when trying to fit models to large
>> datasets using bam. I am working on a 4-core machine, but I think that
>> there may be 2 physical cores that the computer uses as 4 cores in some
>> sense that I don't understand.
>> When I run the bam using makeCluster(3), the model runs on one core. But
>> when I run it on makeCluster(2), top shoes me that three of my cores are
>> taken up to full capacity, and my computer slows down or crashes.
>> How can I get it to truly run on 2 cores?
>> I'm on a thinkpad X230, running ubuntu.
>> R-help at r-project.org mailing list
>> PLEASE do read the posting guide
>> and provide commented, minimal, self-contained, reproducible code.
More information about the R-help