[R] Warning messages while parallel computing

[Bill Dunlap]

Perhaps just try
    source("xx.R")
    invisible(gc()) # do warnings appear after this?
It does seem likely that the sourced file causes the problem, since
parallel::createCluster labels the connections "<host name>:<process
id>" and that label would show up in the warning.

On Thu, Mar 4, 2021 at 11:28 AM Henrik Bengtsson
<henrik.bengtsson using gmail.com> wrote:
>
> Test with:
>
> clusterCall(cl, function() { suppressWarnings(source("xx.R")) })
>
> If the warnings disappear, then the warnings are produced on the
> workers from source():ing the file.
>
> /Henrik
>
> On Thu, Mar 4, 2021 at 10:20 AM Bill Dunlap <williamwdunlap using gmail.com> wrote:
> >
> > The warnings come from the garbage collector, which may be called from
> > almost anywhere.  It is possible that the file that is sourced causes
> > the problem, but if you don't call parallel::stopCluster before
> > removing the cluster object you will get those warnings.
> >
> > > cl <- parallel::makeCluster(3, type="PSOCK")
> > > invisible(gc())
> > > rm(cl)
> > > invisible(gc())
> > Warning messages:
> > 1: In .Internal(gc(verbose, reset, full)) :
> >   closing unused connection 6 (<-Bill-T490:11216)
> > 2: In .Internal(gc(verbose, reset, full)) :
> >   closing unused connection 5 (<-Bill-T490:11216)
> > 3: In .Internal(gc(verbose, reset, full)) :
> >   closing unused connection 4 (<-Bill-T490:11216)
> > >
> > >
> > > cl <- parallel::makeCluster(3, type="PSOCK")
> > > invisible(gc())
> > > parallel::stopCluster(cl)
> > > invisible(gc())
> > > rm(cl)
> > > invisible(gc())
> > >
> >
> > The fact that he got 8 warnings when playing a cluster of size 8 makes
> > me suspect that omitting stopCluster is the problem.
> >
> > -Bill
> >
> > On Thu, Mar 4, 2021 at 10:12 AM Henrik Bengtsson
> > <henrik.bengtsson using gmail.com> wrote:
> > >
> > > I don't think 'parallel' is to blame in this case. Those warnings:
> > >
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 19
> > >
> > > come from base::source()
> > > [https://github.com/wch/r-source/blob/9caddc1eaad1f480283f1e98af34a328699d1869/src/library/base/R/source.R#L166-L244].
> > >
> > > Unless there's a bug in source() that leaves connections open, which
> > > is unlikely, I think there's something in the 'xx.R' script that opens
> > > a connection but doesn't close it.  Possibly multiple times.  A good
> > > check is to see if the same warnings are produced when calling
> > > source("xx.R") sequentially in a for() loop or an lapply() call.
> > >
> > > Hope this helps,
> > >
> > > Henrik
> > >
> > > On Thu, Mar 4, 2021 at 9:58 AM Bill Dunlap <williamwdunlap using gmail.com> wrote:
> > > >
> > > > To avoid the warnings from gc(), call parallel::stopCluster(cl) before
> > > > removing or overwriting cl.
> > > >
> > > > -Bill
> > > >
> > > > On Thu, Mar 4, 2021 at 1:52 AM Shah Alam <dr.alamsolangi using gmail.com> wrote:
> > > > >
> > > > > Hello everyone,
> > > > >
> > > > > I am using the "parallel" R package for parallel computation.
> > > > >
> > > > > Code:
> > > > >
> > > > >    # set number of cores
> > > > >     cl <- makeCluster(8, type = "PSOCK")  # Mac/Linux need to set as "FORK"
> > > > >
> > > > >       # pass functions and objects to the cluster environment and set seed
> > > > >       # all the items exported need to stay in the global environment!!
> > > > >       clusterCall(cl, function() { source("xx.R" )})
> > > > >       clusterExport(cl, list("a", "b", "c", "d",
> > > > >                              "5"))
> > > > >       clusterSetRNGStream(cl, 1)
> > > > >
> > > > > While parallel processing, I receive the following warning signs.  Do I
> > > > > need to ignore these signs or do they potentially slow the whole process?
> > > > >
> > > > >  *  Warning signs:*
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 19
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 18
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 17
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 16
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 15
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 14
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 13
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > >   closing unused connection 12
> > > > >
> > > > > Best regards,
> > > > > Shah Alam
> > > > >
> > > > >         [[alternative HTML version deleted]]
> > > > >
> > > > > ______________________________________________
> > > > > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > > > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > > > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > > > > and provide commented, minimal, self-contained, reproducible code.
> > > >
> > > > ______________________________________________
> > > > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > > > and provide commented, minimal, self-contained, reproducible code.