[R] Re: HOWTO compare univariate binomial glm lrm models which are not nested

[Frank E Harrell Jr]

Jan Verbesselt wrote:
> Dear all,
> 
> Thanks a lot for the input. I will take the considerations into account. 
> 
> Referring to;
> "2 or 3 completely pre-chosen models or you will invalidate inference and
> estimates if you use these comparisons to build a final model"
> 
> The aim is not use the comparisons to build a final model but to select the
> explanatory variable which explains most of the variance or has the best
> predictive ability (p247 10.8 Harrell, 2001).

One procedure that will shed light on this is to bootstrap the ranks of 
the chi-square statistics for competing variables.  I think you will be 
surprised how wide the confidence intervals for the ranks are.  There is 
an example in the Alzola & Harrell document although it is for partial 
chi-squares for competing variables in a single model.

-FH

> 
> I'm comparing variables, which are all related to the remotely sensed water
> content of vegetation, with binary fire occurrence data (1: fire / 0: no
> fire). The aim is to select the water related variable which has the best
> 'performance' (Referring to literature about logistic regression used for
> evaluation of fire danger indices).
> 
> e.g. a lrm model is  lrm(firedata~waterrelated.variable)
> 
> Thanks a lot and best regards,
> Jan
> 
> ***
> "
> In addition to Brian's comment, AIC may be of use.  You can't really use 
> c-index (ROC area) as it is not sensitive enough for comparing two 
> models.  But whatever you use, the bad news is that you can't use the 
> results to compare more than 2 or 3 completely pre-chosen models or you 
> will invalidate inference and estimates if you use these comparisons to 
> build a final model.
> 
> Frank
> "
> ***
> 
> 
> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University