[R] Best performance measure?

[Frank E Harrell Jr]

Noah Silverman wrote:
> Hello,
> 
> I working on a model to predict probabilities.
> 
> I don't really care about binary prediction accuracy.
> 
> I do really care about the accuracy of my probability predictions.
> 
> Frank was nice enough to point me to the val.prob function from the 
> Design library.  It looks very promising for my needs.
> 
> I've put together some tests and run the val.prob analysis.  It produces 
> some very informative graphs along with a bunch of performance measures.
> 
> Unfortunately, I'm not sure which measure, if any, is the "best" one.  
> I'm comparing hundreds of different models/parameter combinations/etc.  
> So Ideally I'd like a single value or two as the "performance measure" 
> for each one.  That way I can pick the "best"  model from all my 
> experiments.
> 
> As mentioned above, I'm mainly interested in the accuracy of my 
> probability predictions.
> 
> Does anyone have an opinion about which measure I should look at??
> (I see Dxy, C, R2, D, U, Briar, Emax, Eavg, etc.)
> 
> Thanks!!
> 
> -N

It all depends on the goal, i.e., the relative value you place on 
absolute accuracy vs. discrimination ability. The Brier score combines 
both and other than interpretability has many advantages.

Frank

> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University