[Rd] Running package tests and not stop on first fail
Hervé Pagès
hpages at fredhutch.org
Tue Nov 8 18:59:37 CET 2016
Thanks Martin.
These changes are great and improve the usefulness of 'R CMD check'.
Especially in the context of the Bioconductor daily builds where
we'll use --no-stop-on-test-error so developers will get a full
picture of all the errors in their package at once.
Cheers,
H.
To provide some context to my special interest for this,
On 11/08/2016 01:34 AM, Martin Maechler wrote:
>>>>>> Hervé Pagès <hpages at fredhutch.org>
>>>>>> on Mon, 7 Nov 2016 14:37:15 -0800 writes:
>
> > On 11/05/2016 01:53 PM, Martin Maechler wrote:
> >>>>>>> Oliver Keyes <ironholds at gmail.com>
> >>>>>>> on Fri, 4 Nov 2016 12:42:54 -0400 writes:
> >>
> >> > On Friday, 4 November 2016, Martin Maechler
> >> > <maechler at stat.math.ethz.ch> wrote:
> >>
> >> >> >>>>> Dirk Eddelbuettel <edd at debian.org <javascript:;>>
> >> >> >>>>> on Fri, 4 Nov 2016 10:36:52 -0500 writes:
> >> >>
> >> >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
> >> >> My > proposed name '--no-stop-on-error' was a quick shot;
> >> >> if | > somebody has a more concise or better "English
> >> >> style" > wording | (which is somewhat compatible with all
> >> >> the other > options you see | from 'R CMD check --help'),
> >> >> | please > speak up.
> >> >>
> >> >> > Why not keep it simple? The similar feature this most
> >> >> > resembles is 'make -k' and its help page has
> >> >>
> >> >> > -k, --keep-going
> >> >>
> >> >> > Continue as much as possible after an > error. While
> >> >> the target that failed, and those that > depend on it,
> >> >> cannot be remade, the other dependencies of > these
> >> >> targets can be processed all the same.
> >> >>
> >> >> Yes, that would be quite a bit simpler and nice in my
> >> >> view. One may think it to be too vague,
> >>
> >> > Mmn, I would agree on vagueness (and it breaks the pattern
> >> > set by other flags of human-readability). Deep familiarity
> >> > with make is probably not something we should ask of
> >> > everyone who needs to test a package, too.
> >>
> >> > I quite like stop-on-error=true (exactly the same as the
> >> > previous suggestion but shaves off some characters by
> >> > inverting the Boolean)
> >>
> >> Thank you, Brian, Dirk and Oliver for these (and some offline)
> >> thoughts and suggestions!
> >>
> >> My current summary:
> >>
> >> 1) I really don't want a --<option-key>=value
> >> but rather stay with logical/binary variables that "express
> >> themselves"... in the same way I strongly prefer
> >>
> >> if (A_is_special) ....
> >> to
> >> if (A_special == TRUE) ....
> >>
> >> for a logical variable A_* . Yes, this is mostly a matter
> >> of taste,.. but related to how R style itself "works"
> >>
> >> 2) Brian mentioned that this is only about ./tests/ tests which
> >> are continued, not about the Examples which are treated separately.
> >> That's why we had contemplated additionally using 'tests' (because that's
> >> the directory name used for unit/regression/.. tests) in the option
> >> name.
> >>
> >> Even though Brian is correct, ideally we *would* want to also influence the
> >> examples' running to *not* stop on a first error.. However that would
> >> need more work, reorganizing how the examples are run and that may not be
> >> worth the pain. However it should be considered a goal in the long run.
>
> > My name is Hervé, and I was not suggesting that what happens with the
> > examples should be changed. I was just preaching consistency (again
> > sorry) between what happens with the examples and what happens with
> > the tests.
>
> Thank you, Hervé and excuse me for not answering more focused on
> what you said.
> I think I do understand what you say (at least by now :-)) and
> agree that consistency is something important and to be strived for,
> also with these options.
>
> > Why not simply change the latter?
> > Do we really need an option to control this?
>
> Very good questions. If the change could be made much better,
> I'd agree we'd not need a new option because the change could be
> considerided uniformly better than the current (R 3.3.2, say) behavior.
> However the change as it is currently, is not good enough to be
> the only option (see below).
>
> > The behavior was changed for the examples a couple of
> > years ago and nobody felt the need to introduce an option
> > to control this at the time.
>
> Yes, that change was made very nicely (not by me) and I'd say
> the result *was* uniformly better than the previous behavior, so
> there did not seem much of a reason to still provide the old behavior.
>
> >> After all that, I tend to remain with the original proposed name. It is at
> >> least easy to pronounce and spell correctly...
>
> > Unless --no-stop-on-error controls both (i.e. examples and tests), and
> > both behave the same way by default, this is a misnomer.
>
> Well, if you choose the option, there *is* no stop on errors anymore
> ... because the examples nowadays never stop the (R CMD check Pkg)
> from running on an error as we've mentioned above.
>
> [[--- Digression on Examples' checking
>
> However / on the other hand: Because of the way the examples
> are run --- efficiently from one single R source file --- it
> is not so easy there, to let them run further: The first error
> from all the examples stops running the example-checks, i.e.,
> the <pkg>-Ex.R script, but at least *does* continue to run the
> next 'R CMD check ...' steps.
>
> To change this without incurring considerable drawbacks, or
> involve a lot of changes does not seem easy:
>
> - If each example would be run in a separate R process, R has to
> be restarted once for every man/*.Rd file.. which easily adds
> one minute or rather several minutes to the total testing time
> (and a I think quite a bit of the checking code had to be re-written).
>
> - Alternatively, all examples are parsed (fail-safe!) first and put together
> into an R list (or environment), and then each run via tryCatch(.).
> This is nicer and more efficient conceptually, but needs even
> more code changes and tends to lose the transparent *-Ex.R -> *-Ex.Rout
> interface we have now.
>
> --- end of digression ---]]
>
> > If this option controls the tests only, it should be more something like
> > --no-stop-on-test-error. If in the long run, another option is added to
> > control the examples, then could be --no-stop-on-example-error.
>
> > But again, maybe all this is not needed at all. Maybe changing what
> > happens with the tests would be enough.
>
> I agree: it would be enough, if the change was better than what we currently have
> (in R-devel): On
> https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176,
> Jan mentions that the patch is simple but suffers a deficiency
> (my wording) in that the error *reporting* is not correct:
>
> If you have three tests/*.R test scripts producing an error,
> still only the first of these is reported (and counted in the
> final 'Status: ..' line), and the package checker has to
> manually look at the tests/*.Rout.fail files to see which of
> the checks failed exactly.... and the reporting of the first
> error is *after* running all the tests/*.R scripts which is also
> far from ideal.
>
>
> Consequently, for now, it does need an option there, and it
> should not be default because of the "inconsistent" report
> output.
>
> I don't mind to add the extra "-test" to the option string,
> i.e., change
> from
> --no-stop-on-error
> to --no-stop-on-test-error
>
> Others?
>
>
> > Cheers,
> > H.
>
> > --
> > Hervé Pagès
>
> > Program in Computational Biology
> > Division of Public Health Sciences
> > Fred Hutchinson Cancer Research Center
> > 1100 Fairview Ave. N, M1-B514
> > P.O. Box 19024
> > Seattle, WA 98109-1024
>
> > E-mail: hpages at fredhutch.org
> > Phone: (206) 667-5791
> > Fax: (206) 667-1319
>
--
Hervé Pagès
Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024
E-mail: hpages at fredhutch.org
Phone: (206) 667-5791
Fax: (206) 667-1319
More information about the R-devel
mailing list