[R] inconsistency in mclapply.....

[akshay kulkarni]

Dear Ivan,
                 I am reopening this thread because there are some new developments. I have switched to ubuntu and have used ropenblas() in ropenblas package to download the latest openblas (it is a version of skylake as you can see below in sessionInfo()). mclapply() is working fine now... My questions:


  1.  I was working on a two core machine just to test the code...tomorrow will be switching to a 48 core machine. Should I again download the latest openblas from ropenblas() or do you suggest testing the native openblas that is bundled in Ubuntu( it was a basic version of openblas..in my two core machine I tried the ropenblas() function without testing the native openblas...just for curiosity) ?
  2.  How do you pass the version of openblas to be installed with ropenblas()? The full file name? or just something like this:
libopenblas_skylakexp-r0.3.23.so  ?

> sessionInfo()
R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS/LAPACK: /opt/OpenBLAS/lib/libopenblas_skylakexp-r0.3.23.so;  LAPACK version 3.11.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods
[8] base

other attached packages:
[1] pbmcapply_1.5.1 forecast_8.21   attempt_0.3.1   imputeTS_3.3

loaded via a namespace (and not attached):
 [1] gtable_0.3.3      compiler_4.3.0    Rcpp_1.0.10       xml2_1.3.4
 [5] scales_1.2.1      lattice_0.20-45   ggplot2_3.4.2     R6_2.5.1
 [9] generics_0.1.3    curl_5.0.1        lmtest_0.9-40     ggtext_0.1.2
[13] tibble_3.2.1      munsell_0.5.0     nnet_7.3-17       timeDate_4022.108
[17] pillar_1.9.0      rlang_1.1.1       quantmod_0.4.22   utf8_1.2.3
[21] urca_1.3-3        quadprog_1.5-8    cli_3.6.1         magrittr_2.0.3
[25] xts_0.13.1        grid_4.3.0        gridtext_0.1.5    lifecycle_1.0.3
[29] nlme_3.1-155      fracdiff_1.5-2    vctrs_0.6.2       glue_1.6.2
[33] tseries_0.10-54   zoo_1.8-12        fansi_1.0.4       colorspace_2.1-0
[37] TTR_0.24.3        stinepack_1.4     pkgconfig_2.0.3


________________________________
From: Ivan Krylov <krylov.r00t using gmail.com>
Sent: Saturday, June 10, 2023 7:50 PM
To: akshay kulkarni <akshay_e4 using hotmail.com>
Cc: R help Mailing list <r-help using r-project.org>
Subject: Re: [R] inconsistency in mclapply.....

On Sat, 10 Jun 2023 13:46:12 +0000
akshay kulkarni <akshay_e4 using hotmail.com> wrote:

>                    What if I switch to Ubuntu?
> I had a similar ( but not exactly the same) problem way back in 2018
> and switching to Ubuntu solved it.

You shouldn't have to switch distros for problems like this one.

What you need to switch is your OpenBLAS build: instead of using
OpenBLAS built with OpenMP support (which isn't even the default
when compiling OpenBLAS), use OpenBLAS built with pthreads support. It's
also likely, but not 100% guaranteed, that setting OMP_NUM_THREADS=1
will avoid the issue by never starting any additional threads for
OpenBLAS purposes. At least some documentation says that
OPENBLAS_NUM_THREADS is ignored on OpenMP builds of OpenBLAS, you
actually have to set OMP_NUM_THREADS=1 for it to take any effect.

I know very little about RHEL, but it seems that you already have a
pthreads build of OpenBLAS installed in
/usr/lib64/libopenblasp-r0.3.15.so; you just need to adjust the path to
the BLAS implementation used by R. If not, install the openblas-threads
package.

Can you use flexiblas
<https://cran.r-project.org/bin/linux/fedora/#blaslapack-switching> to
switch the BLAS implementation to OPENBLAS-THREADS? This should be
possible by using the flexiblas R package or by setting environment
variables
<https://github.com/mpimd-csc/flexiblas#selecting-the-backend-at-runtime>.

That said, I know for sure that on Debian and Ubuntu you get a choice
between libopenblas0-openmp / libopenblas0-pthread /
libopenblas0-serial, and installing any one of these packages except
libopenblas0-openmp should avoid the problem. (And the thread limit
should be set to 1 if you're running your tasks in parallel by other
means.)

--
Best regards,
Ivan

	[[alternative HTML version deleted]]