[Rd] NOTE when detecting mismatch in output, and codes for NOTEs, WARNINGs and ERRORs

[Paul Gilbert]

On 03/26/2014 04:58 AM, Kirill Müller wrote:
> Dear list
>
>
> It is possible to store expected output for tests and examples. From the
> manual: "If tests has a subdirectory Examples containing a file
> pkg-Ex.Rout.save, this is compared to the output file for running the
> examples when the latter are checked." And, earlier (written in the
> context of test output, but apparently applies here as well): "...,
> these two are compared, with differences being reported but not causing
> an error."
>
> I think a NOTE would be appropriate here, in order to be able to detect
> this by only looking at the summary. Is there a reason for not flagging
> differences here?

The problem is that differences occur too often because this is a 
comparison of characters in the output files (a diff). Any output that 
is affected by locale, node name or Internet downloads, time, host, or 
OS, is likely to cause a difference. Also, if you print results to a 
high precision you will get differences on different systems, depending 
on OS, 32 vs 64 bit, numerical libraries, etc.  A better test strategy 
when it is numerical results that you want to compare is to do a 
numerical comparison and throw an error if the result is not good, 
something like

   r <- result from your function
   rGood <- known good value
   fuzz <- 1e-12  #tolerance

   if (fuzz < max(abs(r - rGood))) stop('Test xxx failed.')

It is more work to set up, but the maintenance will be less, especially 
when you consider that your tests need to run on different OSes on CRAN.

You can also use try() and catch error codes if you want to check those.

Paul

>
> The following is slightly related: Some compilers and static code
> analysis tools assign a numeric code to each type of error or warning
> they check for, and print it. Would that be possible to do for the
> anomalies detected by R CMD check? The most significant digit could
> denote the "severity" of the NOTE, WARNING or ERROR. This would further
> simplify (semi-)automated analysis of the output of R CMD check, e.g. in
> the context of automated tests.
>
>
> Best regards
>
> Kirill
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel