[R] net classification improvement?

[Essers, Jonah]

Actually, I don't think I made myself clear and I wrote this late last
night....Sorry. More the issue is that the raw model predictions (from 0
to 1) have no inherent clinical value to them. I.e. They aren't "risk of
disease" or "risk of outcome". They are raw scores that are specific to
each model and are meant to discriminate one disease from another disease.
Trying to compare models is impossible because the NRI requires cutoff
values.  The cutoffs are different for each model.

So, as I've done more reading, it appears the the IRI--Integrated
Discrimination Improvement Index--which is naïve to cutoff values--may be
more what I'm looking for. Does this make sense? I guess I just need a
sanity check.

I have been toying with the PredictABEL package and this seems to like my
data inputs just fine and relies on HMISC and ROCR, both packages I know
well.

Thanks
jonah

On 1/17/12 11:49 AM, "Kevin E. Thorpe" <kevin.thorpe at utoronto.ca> wrote:

>On 01/17/2012 07:16 AM, Essers, Jonah wrote:
>> Thanks for the reply. I think more the issue is whether it can be
>>applied
>> to cross-sectional data. This I'm not sure. This method is heavily cited
>> in the New England Journal of Medicine, but thus far I've only seen it
>> used with longitudinal data.
>
>As I recall, the Pencina et al paper does not suggest it cannot be used
>outside of longitudinal data.  In fact, I don't remember them using
>longitudinal data at all.  So, unless I'm misunderstanding your
>question, I think the function in Hmisc (whose name I always forget)
>should be fine.
>
>>
>> On 1/16/12 10:23 PM, "Kevin E. Thorpe"<kevin.thorpe at utoronto.ca>  wrote:
>>
>>> On 01/16/2012 08:10 PM, Essers, Jonah wrote:
>>>> Greetings,
>>>>
>>>> I have generated several ROC curves and would like to compare the
>>>>AUCs.
>>>> The data are cross sectional and the outcomes are binary. I am testing
>>>> which of several models provide the best discrimination. Would it be
>>>> most
>>>> appropriate to report AUC with 95% CI's?
>>>>
>>>> I have been looking in to the "net reclassification improvement" (see
>>>> below for reference) but thus far I can only find a version in Hmisc
>>>> package which requires survival data. Any idea what the best approach
>>>>is
>>>> for cross-sectional data?
>>>
>>> I believe that the function in Hmisc that does this will also work on
>>> binary data.
>>>
>>>>
>>>> Thanks
>>>>
>>>> Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating
>>>>the
>>>> added predictive ability of a new marker: from area under the ROC
>>>>curve
>>>> to
>>>> reclassification and beyond. Stat Med 2008;27:157-172
>>>>
>>>
>
>
>-- 
>Kevin E. Thorpe
>Biostatistician/Trialist,  Applied Health Research Centre (AHRC)
>Li Ka Shing Knowledge Institute of St. Michael's
>Assistant Professor, Dalla Lana School of Public Health
>University of Toronto
>email: kevin.thorpe at utoronto.ca  Tel: 416.864.5776  Fax: 416.864.3016