[R] Parallel processes collapse into one

[Ivan Krylov]

On Mon, 3 Jun 2019 06:37:46 +0200
Nicola Lunardon <nicola.lunardon.84 using gmail.com> wrote:

> R seems to be running, but simulations do not progress further.

Have you tried message() (or REprintf() in C code) to narrow down the
specific part of the code where simulations stop in their progress?
It's less convenient than a good debugger, but with parallel code it is
sometimes the only way to reproduce the problem and get some
information about it.

> If I run simulations on 16 cores I end up having an R instance with
> CPU usage about 1600%, never experienced such a behaviour.

> BLAS:   /usr/lib/openblas-base/libblas.so.3
> LAPACK: /usr/lib/libopenblasp-r0.2.18.so

OpenBLAS can, indeed, use multiple threads on its own, inside a single
process. Combined with mclapply, this might create a situation when
there are far more threads competing for CPU time than CPU cores
available. Does it help if you set an environment variable such as
OPENBLAS_NUM_THREADS [*] to a number less than or equal to (number of
CPU cores / mc.cores mclapply argument)?

-- 
Best regards,
Ivan

[*]
https://github.com/xianyi/OpenBLAS#setting-the-number-of-threads-using-environment-variables