[Rd] serialization regression in 2.15.0 beta
Prof Brian Ripley
ripley at stats.ox.ac.uk
Wed Mar 28 07:56:58 CEST 2012
On 27/03/2012 22:01, Ben Goodrich wrote:
> In case anyone is concerned that this regression will affect them, the
> code was reverted to the 2.14.x behavior by
>
> ------------------------------------------------------------------------
> r58842 | ripley | 2012-03-26 08:12:43 -0400 (Mon, 26 Mar 2012) | 1 line
> Changed paths:
> M /branches/R-2-15-branch/doc/NEWS.Rd
> M /branches/R-2-15-branch/src/library/parallel/R/unix/forkCluster.R
> M /branches/R-2-15-branch/src/library/parallel/R/unix/mcfork.R
>
> revert to XDR serialization for 2.15.0
> ------------------------------------------------------------------------
But the underlying problem (in non-xdr binary unserialization) is AFAWK
fixed: it was just that at this late stage there was too little time to
test thoroughly before release.
Please test R-devel on your own problem (we haven't: the issue was found
using a different example from elsewhere).
> Thanks,
> Ben
>
>
>> I am experiencing a problem related to serialization behavior in
>> 2.15.0 beta (binary installed from Debian unstable) and 2.16.0 (from
>> svn) that is not present in 2.14.2 (binary from Debian testing).
>>
>> I don't fully understand the problem. Also, I tried but have not yet
>> been able to create a small, self-contained example that reproduces
>> the problem. However, I do have a large, not self-contained example,
>> which requires an alpha version (not yet on CRAN) of the mi package
>> (the mi package on CRAN would not exhibit this issue). Anyone
>> interested in reproducing the problem can follow the readme.txt file
>> in this directory:
>>
>> http://www.columbia.edu/~bg2382/mi/serialization/
>>
>> I track r-devel with git-svn and was able to git bisect to svn commit r58219
>>
>> commit 799102bd9d0266fe89c3120981decf0b1f17ef11
>> Author: ripley<ripley at 00db46b3-68df-0310-9c12-caf00c1e9a41>
>> Date: Sat Jan 28 15:02:34 2012 +0000
>>
>> make use of non-xdr serialization;.
>>
>> although this commit could merely expose the problem rather than cause it.
>>
>> The problem occurs when the FUN called by mclapply() in the parallel
>> package returns a S4 object that contains a slot (called X) that is a
>> large matrix, specifically a "model matrix" similar to that produced
>> by glm(). Some columns of this matrix get corrupted with wrong values
>> (usually zero, but sometimes NaN or 10^300ish), which can be seen by
>> examining X right before FUN returns (to mclapply()'s environment) and
>> comparing to the "same" X after mclapply() returns to the calling
>> environment.
>>
>> Part of svn commit r58219 is this hunk
>>
>> diff --git a/src/library/parallel/R/unix/mcfork.R
>> b/src/library/parallel/R/unix/mcfork.R
>> index 8e27534..4f92193 100644
>> --- a/src/library/parallel/R/unix/mcfork.R
>> +++ b/src/library/parallel/R/unix/mcfork.R
>> @@ -82,7 +82,8 @@ mckill<- function(process, signal = 2L)
>> ## used by mcparallel, mclapply
>> sendMaster<- function(what)
>> {
>> - if (!is.raw(what)) what<- serialize(what, NULL, FALSE)
>> + # This is talking to the same machine, so no point in using xdr.
>> + if (!is.raw(what)) what<- serialize(what, NULL, xdr = FALSE)
>> .Call(C_mc_send_master, what, PACKAGE = "parallel")
>> }
>>
>> Contrary to the comment, I have found that if I specify xdr = TRUE, I
>> get the expected (non-corrupted X slot) behavior in 2.16.0, even
>> though it is forking locally on my 64bit Debian laptop with a little
>> endian i7 processor, whose specs are
>>
>> goodrich at CYBERPOWERPC:/tmp/serialization$ cat /proc/cpuinfo
>> processor : 0
>> vendor_id : GenuineIntel
>> cpu family : 6
>> model : 42
>> model name : Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz
>> stepping : 7
>> microcode : 0x17
>> cpu MHz : 800.000
>> cache size : 6144 KB
>> physical id : 0
>> siblings : 8
>> core id : 0
>> cpu cores : 4
>> apicid : 0
>> initial apicid : 0
>> fpu : yes
>> fpu_exception : yes
>> cpuid level : 13
>> wp : yes
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
>> xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl
>> vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt
>> tsc_deadline_timer xsave avx lahf_lm ida arat epb xsaveopt pln pts dts
>> tpr_shadow vnmi flexpriority ept vpid
>> bogomips : 3990.83
>> clflush size : 64
>> cache_alignment : 64
>> address sizes : 36 bits physical, 48 bits virtual
>> power management:
>>
>> ...
>>
>> processor : 7
>> [same as processor 0]
>>
>> So, to summarize I get the good behavior on R 2.14.2 when using
>> mclapply(), on 2.15.0 beta when using lapply(), and on 2.16.0 using
>> mclapply() iff I patch in xdr = TRUE in sendMaster(). I get the bad
>> behavior on 2.15.0 beta and unpatched 2.16.0 when using mclapply().
>>
>> My session info:
>>
>>> sessionInfo()
>> R version 2.15.0 beta (2012-03-16 r58769)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>
>> locale:
>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
>> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
>> [7] LC_PAPER=C LC_NAME=C
>> [9] LC_ADDRESS=C LC_TELEPHONE=C
>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>>
>> attached base packages:
>> [1] stats4 stats graphics grDevices utils datasets methods
>> [8] base
>>
>> other attached packages:
>> [1] mi_0.9-83 bigmemory_4.2.11 arm_1.5-03 foreign_0.8-49
>> [5] abind_1.4-0 R2WinBUGS_2.1-18 coda_0.14-5 lme4_0.999375-42
>> [9] Matrix_1.0-4 lattice_0.20-0 MASS_7.3-17
>>
>> loaded via a namespace (and not attached):
>> [1] grid_2.15.0 nlme_3.1-103
>>
>> Thanks,
>> Ben
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
--
Brian D. Ripley, ripley at stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
More information about the R-devel
mailing list