[R] bam (mgcv) not using the specified number of cores
s.wood at bath.ac.uk
Mon Aug 25 15:16:53 CEST 2014
oops, I just realised that the fix referred to below is in mgcv 1.8-3 -
not yet on CRAN.
On 25/08/14 13:20, Simon Wood wrote:
> Hi Andrew,
> In some of the shots you sent then top was reporting several cores
> working. I think the problem here may be to do with the way bam is
> parallelized - at present only part of the computation is in parallel -
> the model matrix QR decomposition part. The smoothing parameter
> selection is still single cored (although we are working on that), so if
> you watch top, you'll usually see multi-core and single core phases
> alternating with each other. The strategy works best in n>>p situations
> with few smoothing parameters.
> For the case where you used 31 cores, there was a bug in earlier mgcv
> versions in which it was assumed that when the model matrix is split
> into chunks for processing, each chunk would have more rows than
> columns. If you upgrade to the current mgcv version then this is fixed.
> However using 31 cores is liable to actually be less efficient than
> using fewer cores with the n to p (number of data to number of
> coefficients) ratio that you seem to have. This is because the work
> being done by each core is rather little, so that the overhead of
> stitching the cores' work back together becomes too high. Using
> 'use.chol=TRUE' would reduce the overheads here (although it uses a
> slightly less stable algorithm than the default).
> On 22/08/14 06:13, Andrew Crane-Droesch wrote:
>> Hi Simon,
>> (resending with all images as imgur so as to not bounce from list)
>> Thanks for the reply. I've tried to reproduce the error, but I don't
>> know how to show output from `top` any other way than with screenshots,
>> so please excuse that.
>> Here are screenshots of what happens when I run with two
>> and three
>> cores. In the former, it seems to be working on one core, and in the
>> latter, it appears to be working on three. When reproducing the error,
>> I'm getting behavior that isn't entirely consistent -- sometimes it
>> "behaves" and operates on the asked-for number of cores, and other times
>> I'm also attaching a screenshot
>> showing terminal output from a remote cluster when I run my full model
>> (N=67K) rather than a subset (N=7K) -- I get that error "Error in
>> qr.qty(qrx, f) : right-hand side should have 60650 not 118451 rows ..."
>> I suppose this is a memory overload problem? Any suggestions on how to
>> get bam to not call for more memory than the node has available would be
>> welcome, though I suspect that is a supercomputing problem rather than a
>> mgcv problem. I don't know much about memory management, except that R
>> doesn't do it explicitly.
>> sessionInfo() for local machine:
>> 1> sessionInfo()
>> R version 3.0.2 (2013-09-25)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>  LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
>>  LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
>>  LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
>>  LC_PAPER=en_US.UTF-8 LC_NAME=C
>>  LC_ADDRESS=C LC_TELEPHONE=C
>>  LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>> attached base packages:
>>  parallel stats graphics grDevices utils datasets methods
>>  base
>> other attached packages:
>>  mgcv_1.7-26 nlme_3.1-111
>> loaded via a namespace (and not attached):
>>  grid_3.0.2 lattice_0.20-23 Matrix_1.1-4
>> On 08/21/2014 04:29 PM, Simon Wood wrote:
>>> Hi Andrew,
>>> Could you provide a bit more information, please. In particular the
>>> results of sessionInfo() and the code that caused this weird behaviour
>>> (+ an indication of dataset size).
>>> On 21/08/14 12:53, Andrew Crane-Droesch wrote:
>>>> I am getting strange behavior when trying to fit models to large
>>>> datasets using bam. I am working on a 4-core machine, but I think that
>>>> there may be 2 physical cores that the computer uses as 4 cores in some
>>>> sense that I don't understand.
>>>> When I run the bam using makeCluster(3), the model runs on one core.
>>>> when I run it on makeCluster(2), top shoes me that three of my cores
>>>> taken up to full capacity, and my computer slows down or crashes.
>>>> How can I get it to truly run on 2 cores?
>>>> I'm on a thinkpad X230, running ubuntu.
>>>> R-help at r-project.org mailing list
>>>> PLEASE do read the posting guide
>>>> and provide commented, minimal, self-contained, reproducible code.
Simon Wood, Mathematical Science, University of Bath BA2 7AY UK
+44 (0)1225 386603 http://people.bath.ac.uk/sw283
More information about the R-help