[Rd] nested parallel workers

[Valerie Obenchain]

Hi Simon,

I'm having trouble with nested parallel workers, specifically, forking
inside socket connections.

When mclapply is called inside a SOCK, PSOCK or FORK worker I get an
error in unserialize().

cl <- makeCluster(1, "SOCK")

fun = function(i) {
   library(parallel)
   mclapply(1:2, sqrt)
}

Failure occurs after multiple calls to clusterApply:

 > clusterApply(cl, 1, fun)
[[1]]
[[1]][[1]]
[1] 1

[[1]][[2]]
[1] 1.414214

 > clusterApply(cl, 1, fun)
[[1]]
[[1]][[1]]
[1] 1

[[1]][[2]]
[1] 1.414214

 > clusterApply(cl, 1, fun)
Error in unserialize(node$con) : error reading from connection


This example is from Martin and may be a different problem.

~/tmp >cat test1.R
## like mclapply
## should run 'forever' but terminates semi-randomly
library(parallel)
children <- parallel:::children

while (TRUE) {
     n <- 8            ## n == dectectCores()
     jobs <- lapply(seq_len(n), function(i) mcparallel(Sys.sleep(20)))
     mccollect(children(jobs), FALSE)
     parallel:::mckill(children(jobs), tools::SIGTERM)
     leni <- length(mccollect(children(jobs)))
     message("leni: ", leni)
}

~/tmp >R-dev --vanilla --slave -f test1.R
leni: 6
leni: 7
leni: 7
leni: 7
leni: 7
leni: 7
leni: 7
leni: 7
leni: 8
leni: 7
leni: 7
leni: 7
~/tmp >


Thanks.
Valerie


 > sessionInfo()
R Under development (unstable) (2015-03-18 r68009)
Platform: x86_64-unknown-linux-gnu (64-bit)
Running under: Fedora 21 (Twenty One)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods
[8] base

loaded via a namespace (and not attached):
[1] snow_0.3-13


-- 
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, Seattle, WA 98109

Email: vobencha at fredhutch.org
Phone: (206) 667-3158