[Rd] issue with parallel package

[Simon Urbanek]

On May 21, 2014, at 1:17 PM, Josef Leydold <josef.leydold at wu.ac.at> wrote:

> Dear maintainers of the parallel package,
> 
> I ran into an issue with the parallel package in R-3.1.0.
> 
> The following code prints the message "NULL!" quite a lot.
> 
> library(parallel)
> for (n in 1:1000) {
>       p <- mcparallel(sqrt(n))
>       res <- mccollect(p, wait=FALSE, timeout=1000)
>       mccollect(p)
>       if (is.null(res)) cat(n,"  NULL!\n")
> }
> 
> 
> It does not happen in R-3.0.3.
> (To be exact, I have checked out R-devel revision r65130
> for my current tests.)
> 
> I found that this behavior is introduced in revision r65143.
> 
> When I insert
>   Sys.sleep(0.1)
> in the loop then this behavior disappears in R-3.1.0.
> 
> There is also another difference between R-3.1.0 and R-3.0.3:
> When the line
>   mccollect(p)
> is replaced by
>   mccollect(p, wait=FALSE, timeout=1000)
> then in R-3.1.0 some child processes are not terminated and some
> pipes remain open. This does not happen in R-3.0.3.
> 
> Remark:
> If the line
>   mccollect(p)
> is omitted at all then "NULL!" is never printed but
> R aborts after 510 iterations of the loop with the message
> "Error in mcfork(detached) : unable to create a pipe".
> Then none of the child processes is terminated and
> number of possible open pipes is exhausted.
> 
> Is this behavior intended or a bug?
> 

As noted in the follow-up this is absolutely expected. wait=FALSE merely checks if the result is available - if it's not at that point then NULL is returned - that's what you see and is exactly what one would expect. Analogously, if you don't collect the results then the child processes will be still hanging around waiting for you to collect the results and therefore you'll run out of descriptors. The fact that it worked in 3.0.3 is just a coincidence that it used more time managing the children and checking their status before you got to your attempted collect so the result was more likely to be already available. Because 3.1.0 is more efficient (it no longer manages children manually but rather uses signal-based cleanup), it comes faster to the collect point and the results may not be available yet.

Cheers,
Simon



> 
>> version
>               _                           
> platform       x86_64-pc-linux-gnu         
> arch           x86_64                      
> os             linux-gnu                   
> system         x86_64, linux-gnu           
> status                                     
> major          3                           
> minor          1.0                         
> year           2014                        
> month          04                          
> day            10                          
> svn rev        65387                       
> language       R                           
> version.string R version 3.1.0 (2014-04-10)
> nickname       Spring Dance                
> 
> 
> best whishes
> Josef
> 
> -- 
> 
> 
> -----------------------------------------------------------------------------
> 
> Josef Leydold
> 
> Institute for Statistics and Mathematics
> WU (Vienna University of Economics and Business)
> 
> Welthandelsplatz 1, Building D4, 1020 Vienna, Austria
> 
> Phone +43 1 31336 4695  FAX +43 1 31336 904695  email josef.leydold at wu.ac.at
> 
> -----------------------------------------------------------------------------
> Alles Unglueck kam daher, dass die Denkenden nicht mehr handeln konnten,
> und die Handelnden keine Zeit mehr fanden zu denken.       (Marlen Haushofer)
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>