[R] Fw: Logistic regresion - Interpreting (SENS) and (SPEC)

[Frank E Harrell Jr]

Gad Abraham wrote:
> Frank E Harrell Jr wrote:
>> Gad Abraham wrote:
>>>> This approach leaves much to be desired.  I hope that its 
>>>> practitioners start gauging it by the mean squared error of 
>>>> predicted probabilities.
>>>
>>> Is the logic here is that low MSE of predicted probabilities equals a 
>>> better calibrated model? What about discrimination? Perfect calibration 
>>
>> Almost.  I was addressed more the wish for the use of strategies that 
>> maximize precision while keeping bias to a minimim.
>>
>>> implies perfect discrimination, but I often find that you can have two 
>>
>> That doesn't follow.  You can have perfect calibration in the large 
>> with no discrimination.
> 
> I'm not sure I understand: if you have perfect calibration, so that you 
> correctly assign the probability Pr(y=1|x) to each x, doesn't it follow 
> that the x will also be ranked in correct order of probability, which is 
> what the AUC is measuring?

You can have a prediction model that assigns Pr(y=1|x) to a range of 
0.45 to 0.55 such that the probabilities are perfectly accurate, but the 
ROC area is 0.6.
> 
>>
>>> competing models, the first with higher discrimination (AUC) and 
>>> worse calibration, and the the second the other way round. Which one 
>>> is the better model?
>>
>> I judge models on the basis of both discrimination (best measured with 
>> log likelihood measures, 2nd best AUC) and calibration.  It's a 
>> two-dimensional issue and we don't always know how to weigh the two. 
>> For many purposes calibration is a must.  In those we don't look at 
>> discrimination until calibration-in-the-small is verified at high 
>> resolution.
> 
> By "log likelihood measures" do you mean likelihood-ratio tests?

I mean generalized R^2, log-likelihood, or the adequacy index in my book.

Frank

> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University