[Rd] R CMD check recursive copy of tests/

[Henrik Bengtsson]

Hi.

On 8/30/07, Martin Maechler <maechler at stat.math.ethz.ch> wrote:
> >>>>> "HenrikB" == Henrik Bengtsson <hb at stat.berkeley.edu>
> >>>>>     on Wed, 29 Aug 2007 15:11:17 -0700 writes:
>
>     >> From NEWS of R v2.6.0 devel:
>     HenrikB>  o R CMD check now does a recursive copy on the
>     HenrikB> 'tests' directory.
>
>     HenrikB> However, R CMD check does not run *.R scripts in
>     HenrikB> such subdirectories (as I thought/hoped for),
>
>  a pretty bold hope ;-)
>
>     HenrikB> only those directly under tests/, This may or may not be
>     HenrikB> intentional.
>
> intentional I'd say: I did not implement it, but it seems much
> more logical to keep the previous rule: All *.R files in
> ./tests/ are run <period>
> Subdirectories can be useful for organization, notably storing
> test data.  I don't think it's a good idea to use so very many test files
> that you need subdirectories, unless maybe you are thinking
> about unit tests; and then, see below.

Examples of subdirectories (some overlapping) are:

units/ - tests of minimal code modules
integration/ - tests of integrating the above units
system/ - "real-world" scenarios/use cases

requirements/ - every requirement should have at least on test.
bugs/ - every bug fix should come with a new test.
regression/ - every update should have a regression test to validate
backward compatibility etc.

robustness/ - Testing the robustness of estimators against outliers as
well as extreme parameter settings.
validation/ - validation of numeric results compared with alternative
implementations or summaries.

benchmarking/ - actually more measuring time, but can involve
validation that a method is faster than an alternative.
crossplatform/ - validate correctness across platforms.
torture/ - pushing the limits.


>
>
>     HenrikB> If true, maybe the above should be clarified as:
>
>     HenrikB>  o R CMD check now does a recursive copy on the
>     HenrikB> 'tests' directory for the purpose of provided data
>     HenrikB> files for input.  Test scripts still has to be
>     HenrikB> directly under tests/ to be run.
>
>     HenrikB> Just a comment
>
> Thanks for the comment.
>
> Many other people have complained that 'NEWS' is too verbose
> already, and hence too much to be read by an average useR, even programmeR.
> The point about 'NEWS' is to mention things that are *changed*.
> Now since the behavior did *not* change, we definitely should
> not make NEWS longer just to say that.
>
>  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> A note about ./tests/ , directories, etc :
>
> At the useR!2007 meeting three weeks ago, there was an
> interesting discussion --- I think after the "Regulated
> Environments / R Certification" session {{and those interested
> there should look at http://www.r-project.org/certification.html}} ---

I ran in to that last week; I think that is a great and important
initiative.  Now I hear that people use R for development and then
reimplement in SAS to please FDA.

> on the broader topic of 'R validation' and testing etc,
> where one of my short contributions was to acknowledge that
> "yes, it would be great if people wrote much more
>  regression/unit test code, and I would welcome if these were
>  collected, and put into daily testing state etc etc"
>
> *Contrary* to the <pkg>/tests/  concept,
> one thing I think we (R developers at large) should consider and
> often favor, is the idea that you want to run the tests also at
> run time and not just install time
> {{and part of that is also desirable for "base R"}}
>
> About a week agao, I have re-organized the
> RUnit-based unit regression tests for two Rmetrics R packages to
> achieve the following goals :
>
> 1) tests should be auto-run via 'R CMD check <pkg>'
> 2) tests should be easily re-runnable by a user who only
>    got the *binary* package {e.g. a typical Windows user}
> 3) tests should also be easily runnable by the package developer
>    without need to run the full R CMD check.
>
> where "1)" and "2)" where really primary goals.
>
> I've basically started from what is written (I think mainly by
> Tony Plate and Gregor Gorjanc) in the R Wiki section
>    http://wiki.r-project.org/rwiki/doku.php?id=developers:runit
> and Gregor Gojanc's example in CRAN package  'gdata'
>
> but then really also found I needed to change things a bit to
> make '2)' more convenient.
>
> The basic idea is:
>
>  ./tests/doRunit.R   is a setup file  {which will not be installed}
>
>  ./inst/unitTests/   contains all the meat {and *is* installed}
> with
>  ./inst/unitTests/runTests.R   calls all the other unit tests
>  ./inst/unitTests/runit*.R
>
> where the idea is the user of the *installed* package
> can easily run the tests again,
> hereby **validating** her installation of the package:
>
> E.g., for the not-yet released version of fCalendar,
>
> > source(system.file("unitTests/runTests.R", package = "fCalendar"))
>
>  RUnit 0.4.17 loaded.
> Now have RUnit Test Suite 'testSuite' for package 'fCalendar' :
>
> ......
>
> Consider doing
>           tests <- runTestSuite(testSuite)
>
> and later
>           printTextProtocol(tests)

I think these are all great ideas/suggests/implementation.  As
important as design and implementation, testing (and bug reporting)
deserves a higher status.

It often the case that it is not feasible to have all tests ran by R
CMD check.  For instance, I have system tests that take literally
hours or even days to run.  I also know that because of their
integrated build/check system, Bioconductor have a rule of thumb that
it shouldn't take more than 5 minutes for R CMD check to finish
[http://wiki.fhcrc.org/bioc/Package_Guidelines#time-requirements].
Then having test cases installed allows you to still include such
tests and run them at other times (run time).

I would like to add that it can also be useful to have the option tp
ask an end-user that reports bugs/problems to run some standard tests
(that might not have been ran by R CMD check).

Cheers

Henrik

>
> -- -- -- -- --
>
> The development version is available via subversion (svn) or also simple
> web tools from  https://svn.r-project.org/Rmetrics/trunk/fCalendar/
>
> Martin Maechler, ETH Zurich
>