[R] Fw: Logistic regresion - Interpreting (SENS) and (SPEC)
Frank E Harrell Jr
f.harrell at vanderbilt.edu
Thu Oct 16 13:49:55 CEST 2008
Gad Abraham wrote:
> Frank E Harrell Jr wrote:
>> Gad Abraham wrote:
>>>> This approach leaves much to be desired. I hope that its
>>>> practitioners start gauging it by the mean squared error of
>>>> predicted probabilities.
>>> Is the logic here is that low MSE of predicted probabilities equals a
>>> better calibrated model? What about discrimination? Perfect calibration
>> Almost. I was addressed more the wish for the use of strategies that
>> maximize precision while keeping bias to a minimim.
>>> implies perfect discrimination, but I often find that you can have two
>> That doesn't follow. You can have perfect calibration in the large
>> with no discrimination.
> I'm not sure I understand: if you have perfect calibration, so that you
> correctly assign the probability Pr(y=1|x) to each x, doesn't it follow
> that the x will also be ranked in correct order of probability, which is
> what the AUC is measuring?
You can have a prediction model that assigns Pr(y=1|x) to a range of
0.45 to 0.55 such that the probabilities are perfectly accurate, but the
ROC area is 0.6.
>>> competing models, the first with higher discrimination (AUC) and
>>> worse calibration, and the the second the other way round. Which one
>>> is the better model?
>> I judge models on the basis of both discrimination (best measured with
>> log likelihood measures, 2nd best AUC) and calibration. It's a
>> two-dimensional issue and we don't always know how to weigh the two.
>> For many purposes calibration is a must. In those we don't look at
>> discrimination until calibration-in-the-small is verified at high
> By "log likelihood measures" do you mean likelihood-ratio tests?
I mean generalized R^2, log-likelihood, or the adequacy index in my book.
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
More information about the R-help