[R] About Mcneil Hanley test for a portion of AUC!
Frank E Harrell Jr
f.harrell at vanderbilt.edu
Thu Jun 12 18:27:02 CEST 2008
Dukka k.c. wrote:
> Dear all
>
> I am trying to compare the performances of several methods using the AUC0.1
> and
> not the whole AUC. (meaning I wanted to compare to AUC's whose x axis only
> goes to
> 0.1 not 1)
>
> I came to know about the Mcneil Hanley test from Bernardo Rangel Tura
> and I referred to the original paper for the calculation of "r" which is an
> argument of the function
> cROC. I can only find the value of "r" for the whole AUC's .
>
>
>
>
>> seROC<-function(AUC,na,nn){
>> a<-AUC
>> q1<-a/(2-a)
>> q2<-(2*a^2)/(1+a)
>> se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
>> se
>> }
>>
>> cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
>> se1<-seROC(AUC1,na1,nn1)
>> se2<-seROC(AUC2,na2,nn2)
>>
>> sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
>> zad<-(AUC1-AUC2)/sed
>> p<-dnorm(zad)
>> a<-list(zad,p)
>> a
>
> Could somebody kindly suggest me how to calculate the value of "r" or
> some ways to calculate the statistical significance measure for the
> differences of auc for a part of
> the curve like AUC0.1.
>
> Thank You
The ROC area is not a sensitive enough measure for comparing two
competing predictors. Its power is too low. See for example the
following papers. Note that Pencina et al's approach is now in the
Hmisc package (function improveProb; documentation to be coming soon).
Likelihood ratio tests are even more powerful.
@Article{pen08eva,
author = {Pencina, Michael J. and {D'Agostino Sr}, Ralph B. and
{D'Agostino Jr}, Ralph B. and Vasan, Ramachandran S.},
title = {Evaluating the added predictive ability of a new marker:
{From} area under the {ROC} curve to reclassification and beyond},
journal = Stat in Med,
year = 2008,
volume = 27,
pages = {157-172},
annote = {discrimination;model performance;AUC;C-index;risk
prediction;biomarker;small differences in ROC area can still be very
meaningful;example of insignificant test for difference in ROC areas
with very significant results from new method;Yates' discrimination
slope;reclassification table;limiting version of this based on whether
and amount by which probabilities rise for events and lower for
non-events when compare new model to old;comparing two models}
}
@Article{coo07use,
author = {Cook, Nancy R.},
title = {Use and misues of the receiver operating characteristic
curve in risk prediction},
journal = {Circulation},
year = 2007,
volume = 115,
pages = {928-935},
annote = {reclassification table;problems with c index;problems
with ROC area;example of large change in predicted risk in
cardiovascular disease with tiny change in ROC area;possible limits to c
index when calibration is perfect;importance of calibration accuracy and
changes in predicted risk when new variables are added}
}
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
More information about the R-help
mailing list