[Rd] Brainstorm: Alpha and Beta testing of R versions

[Andrew Robinson]

Hi Martin,

On Fri, Nov 04, 2005 at 09:58:47AM +0100, Martin Maechler wrote:
> [Mainly for R-foundation members; but kept in public for general
>  brainstorming...]

I'll take up the invitation to brainstorm.

As a user of R for a number of years, I'd really like to perform some
useful service.  I use a relatively obscure platform (FreeBSD) and I
can compile code.  I'd like to think that I'm in the target market for
beta testing :).  But, I'm timid.  I do not feel, in general, that R
core welcomes bug reports.

I think that there are several things that could be tried to encourage
more, and more useful, bug reports.

1) Put the following text on the *front page* of the tracking system, so
   that it is seen before the reader clicks on "New Bug Report":

"Before submitting a bug report, please read Chapter `R Bugs' of `The
R FAQ'. It describes what a bug is and how to report a bug.

If you are not sure whether you have observed a bug or not, it is a
good idea to ask on the mailing list R-Help by sending an e-mail to
r-help at stat.math.ethz.ch rather than submitting a bug report."

(BTW is this true also for alpha/beta testing?)

2) Try to use the structure of the reporting page to prompt good
   reporting.  On the report page, summarize the key points of
   identifying and reporting a bug in a checklist format.  Maybe even
   insist that the boxes be checked before allowing submission.
   Include seperate text boxes for description and sample code, to
   suggest that sample code is valued.

3) On either or both pages (and in FAQ), explain that thoughtful bug
   reports are valued and appreciated.  Further, explain that bug
   reports that do not follow the protocol are less valuable, and take
   more time.

4) Add checkboxes to the report page for alpha/beta.  (I suggest this
   for the purposes of marketing, not organization.)

5) On the report page, include hyperlinks to archived bug reports that
   were good.  Do likewise with some artificial bug reports that are
   bad.

6) Add an intermediate, draft step for bug submission, to allow
   checking.  If possible, include as part of this step an automated
   pattern matching call that identifies similarly texted bug reports,
   provides links to the reports, and invites a last-minute cross-check.

7) Keep a list of people who report useful bugs in alpha/beta phase on
   the website.  Many academics could point to it as evidence of
   community service.

> In order to discourage an increased number of non-bug reports we
> may have to also open a "hall of shame" though...

8) I'm sure that you're being ironic!  But I will take the point
   seriously, for what it's worth.  I think that humiliating
   submitters who haven't followed the protocol is deleterious.  It
   seems like almost every month we see someone get slapped on the
   wrist for not doing something the right way.  Of course, it's
   frustrating that people aren't following the posting guide.  But,
   why is that?  Where is the breakdown?  It might be interesting to
   try some follow-up (an exit interview!). If someone has failed to
   follow the protocol, perhaps we should try to find out why it was
   confusing, or if they just ignored it.

   The R-core is surrounded by, and serves, a community that comprises
   people who are not sufficiently good at what R-core does to be
   invited in to R-core. But, we're clearly interested in what R-core
   produces.  Please don't assume that bug submissions that do not
   follow the R protocol are the consequence of deliberate
   malfeasance.  

   To paraphrase Ian Fleming: Once is happenstance.  Twice is
   incompetence.  The third time, Mr. Bond, is enemy action. So, ...

9) Publicly thank bug reporters whether their reports are useful or
   not.  I just googled 'R-devel thank' and you figure prominently,
   Martin :).

Cheers

Andrew
-- 
Andrew Robinson
Senior Lecturer in Statistics                       Tel: +61-3-8344-9763
Department of Mathematics and Statistics            Fax: +61-3-8344-4599
University of Melbourne, VIC 3010 Australia
Email: a.robinson at ms.unimelb.edu.au    Website: http://www.ms.unimelb.edu.au