[R] Computing Confidence Intervals for AUC in ROCR Package

[David Winsemius]

On Jan 22, 2010, at 8:31 AM, Frank E Harrell Jr wrote:

> David Winsemius wrote:
>> On Jan 22, 2010, at 3:53 AM, Na'im R. Tyson wrote:
>>> Dear R-philes,
>>>
>>> I am plotting ROC curves for several cross-validation runs of a  
>>> classifier (using the function below).  In addition to the average  
>>> AUC, I am interested in obtaining a confidence interval for the  
>>> average AUC.  Is there a straightforward way to do this via the  
>>> ROCR package?
>> You should probably contact the authors. When I tried using that  
>> package a few weeks ago, several of the annotation features were  
>> broken. I contacted the author who said there had been problems  
>> after converting to S4 method. He also said there would be a fix  
>> but not immediately. There has been a release since that time and I  
>> tried it,  but it did not appear to fix the problems I encountered.  
>> All I was able to get were very simple ROC curves without any  
>> confidence intervals or marking of levels. I ended up turning to  
>> the Epi package for what I needed ( but I did not need confidence  
>> intervals so cannot comment on that aspect.)
>
> I'm wondering what was broken with the S3 implementation that made  
> them change to S4.

I was typing from memory and may not have conveyed accurately what was  
in the message. He mention changing versions but my attribution of  
that problem as switching from S3 to S4 methods seems to have been a  
manufactured memory. Furthermore, on loading the package in its  
current form, I am no longer having the problems I earlier experienced.

So now my question to Tyson would be, what you were hoping to see with  
your request for confidence intervals? The "spread estimate" feature  
seems to have been fixed in version 1.0-4.

>
> Frank
>
> -- 
> Frank E Harrell Jr   Professor and Chairman        School of Medicine
>                     Department of Biostatistics   Vanderbilt  
> University

David Winsemius, MD
Heritage Laboratories
West Hartford, CT