[Rd] R CMD build then check fails on R-devel due to serialization version

[Tomas Kalibera]

An update on the adoption of serialization format 3:

R 3.5.0 (released yesterday) supports serialization format 3, but the 
default is still format 2 to make the transition easier.

It is expected that the default will soon be changed to 3 in R-devel. 
Packages thus must not depend on what is the default serialization 
format (several packages used to have hard-coded assumptions about that 
the format was 2).

As of R 3.5.0, one can test the effect of the expected change to 
serialization format 3 by setting environment variables 
"R_DEFAULT_SAVE_VERSION=3" and "R_DEFAULT_SERIALIZE_VERSION=3". One can 
also set these to 2 to ensure the default format is 2 even after the 
expected change.

Packages that include serialization files in format 3 in their sources 
cannot be used by older versions of R than 3.5.0, so they must declare a 
dependency on R>=3.5.0. Package authors need to make sure of this, but 
they only need to worry if they added serialization files explicitly - 
package meta-data will still be saved in format 2 automatically. Note 
that the current release, R 3.5.0, has the default at 2, so no extra 
precaution is needed when creating such serialize files using the 
current release 3.5.0, but authors will have to be careful when creating 
such files using even newer versions of R.

As a safeguard, "R CMD build" automatically adds a dependency on 
R>=3.5.0 when it detects a serialization file in format 3, but package 
authors should not depend on this, because the detection is not and can 
not be fully precise. As another safeguard, "R CMD check --as-cran" will 
reject packages with serialization format 3 unless they have a 
dependency on at least R>=3.5.0.

This should give enough options also for testing frameworks based on 
running build&"check --as-cran" using the same version of R (most 
packages don't include serialize files at all, R CMD build would 
otherwise add dependency on R>=3.5.0 to make "R CMD check --as-cran" 
happy, one could also set R_DEFAULT_SAVE_VERSION=2 and 
R_DEFAULT_SERIALIZE_VERSION=2). These frameworks could also be tested 
before the change by running with R_DEFAULT_SAVE_VERSION=3 and 
R_DEFAULT_SERIALIZE_VERSION=3.

Best
Tomas

On 01/13/2018 01:35 AM, Tomas Kalibera wrote:
> To reduce difficulties for people relying on automated tests set up to 
> build&"check --as-cran" using R-devel (e.g. travis-ci), the default 
> serialization version has been temporarily switched back to 2. Thank 
> you for your patience - according to svn history, the last change of 
> the serialization format happened 16 years ago, and unsurprisingly 
> some practices that developed since did not anticipate such change and 
> have to be adapted.
>
> CRAN is now protected against packages containing serialized files in 
> format 3 (which not only is not readable by 3.4.x and older, but could 
> still change - the 'devel' in 'R-devel'). These new checks have to 
> stay but we are looking at improving package-maintainer-friendliness. 
> It turned out more difficult than just 1-2 days, hence the temporary 
> switch back to version 2.
>
> Best
> Tomas
>
> On 01/11/2018 02:47 PM, luke-tierney at uiowa.edu wrote:
>> As things stand now, package tarballs with vignettes that are built
>> with R-devel will not install in R 3.4.x, so CRAN can't accept them
>> and someone running R CMD check --as-cran should be told that. A
>> WARNING is appropriate.
>>
>> Most likely what will change soon is that build/version.rds will be
>> saved with serialization version = 2 and this warning will not be
>> triggered just by having a vignette. It will still be triggered by
>> data files serialized with R-devel's default version = 3.
>>
>> Please do remember that the 'devel' in R-devel means exactly that:
>> things will at times be unstable. There are currently a lot of balls
>> flying around with changes in R-devel and also Biocontuctor, and the
>> CRAN maintainers are working hard to keep things all up in the
>> air. Please be patient.
>>
>> Best,
>>
>> luke
>>
>> On Thu, 11 Jan 2018, Jim Hester wrote:
>>
>>> This change poses difficulties for automated build systems such as
>>> travis-ci, which is widely used in the R community. In particular
>>> because this is a WARNING and not a NOTE this causes all R-devel
>>> builds with vignettes to fail, as the default settings fail the build
>>> if R CMD check issues a WARNING.
>>>
>>> The simplest change would be for R-core to change this message to be a
>>> NOTE rather than a WARNING, the serialization could still be tested
>>> and there would be a check against vignettes built with R-devel, but
>>> it would not cause these builds to fail.
>>>
>>> On Wed, Jan 10, 2018 at 3:52 PM, Duncan Murdoch
>>> <murdoch.duncan at gmail.com> wrote:
>>>> On 10/01/2018 1:26 PM, Neal Richardson wrote:
>>>>>
>>>>> Hi,
>>>>> Since yesterday I'm seeing `R CMD check --as-cran` failures on the
>>>>> R-devel daily build (specifically, R Under development (unstable)
>>>>> (2018-01-09 r74100)) for multiple packages:
>>>>>
>>>>> * checking serialized R objects in the sources ... WARNING
>>>>> Found file(s) with version 3 serialization:
>>>>> ‘build/vignette.rds’
>>>>> Such files are only readable in R >= 3.5.0.
>>>>> Recreate them with R < 3.5.0 or save(version = 2) or 
>>>>> saveRDS(version =
>>>>> 2) as appropriate
>>>>>
>>>>> As far as I can tell, revision 74099
>>>>>
>>>>> (https://github.com/wch/r-source/commit/d9530001046a582ff6a43ca834d6c3586abd0a97), 
>>>>>
>>>>> which changes the default serialization format to 3, clashes with
>>>>> revision 73973
>>>>> (https://github.com/wch/r-source/commit/885764eb74f2211a547b13727f2ecc5470c3dd00), 
>>>>>
>>>>> which checks that serialized R objects are _not_ version 3. It seems
>>>>> that with the current development version of R, if you `R CMD build`
>>>>> and then run `R CMD check --as-cran` on the built package, it will
>>>>> fail.
>>>>>
>>>>
>>>> I think the message basically says:  don't do that.  You should 
>>>> build with
>>>> R-release for now.  You always need to check with R-devel, so life is
>>>> complicated.
>>>>
>>>> If you build with R-devel without forcing the old format, nobody using
>>>> R-release will be able to use your tarball.
>>>>
>>>> Eventually I guess the new format will be accepted by CRAN, but it 
>>>> will
>>>> likely be a while:  nobody expects everyone to instantly upgrade to 
>>>> a new R
>>>> release, let alone to an unreleased development version.
>>>>
>>>> Presumably that particular file (build/vignette.rds) could be 
>>>> automatically
>>>> built in the old format for now, but the new format needs testing, 
>>>> so it
>>>> makes sense to me to leave it as a default, even if it makes it more
>>>> complicated to submit a package to CRAN.
>>>>
>>>> Duncan Murdoch
>>>>
>>>>
>>>> ______________________________________________
>>>> R-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>