[R] How do I compare 47 GLM models with 1 to 5 interactions and unique combinations?

[Frank Harrell]

Ruben, I'm not sure you are understanding the ramifications of what Bert
said.  In addition you are making several assumptions implicitly:

1. model selection is needed (vs. fitting the full model and using
shrinkage)
2. model selection works in the absence of shrinkage
3. model selection can find the "right" model and the features selected
would be the same features selected if the data were slightly perturbed or a
new sample taken
4. AIC tells you something that P-values don't (unless using structured
multiple degree of freedom tests)
5. parsimony is good

None of these assumptions is true.  Model selection without shrinkage
(penalization) seems to offer benefits but this is largely a mirage.

Frank


Rubén Roa wrote
> 
> -----Mensaje original-----
> De: Bert Gunter [mailto:gunter.berton@] 
> Enviado el: jueves, 26 de enero de 2012 21:20
> Para: Rubén Roa
> CC: Ben Bolker; Frank Harrell
> Asunto: Re: [R] How do I compare 47 GLM models with 1 to 5 interactions
> and unique combinations?
> 
> On Wed, Jan 25, 2012 at 11:39 PM, Rubén Roa <rroa@> wrote:
>> I think we have gone through this before.
>> First, the destruction of all aspects of statistical inference is not at
>> stake, Frank Harrell's post notwithstanding.
>> Second, checking all pairs is a way to see for _all pairs_ which model
>> outcompetes which in terms of predictive ability by -2AIC or more. Just
>> sorting them by the AIC does not give you that if no model is better than
>> the next best by less than 2AIC.
>> Third, I was not implying that AIC differences play the role of
>> significance tests. I agree with you that model selection is better not
>> understood as a proxy or as a relative of significance tests procedures.
>> Incidentally, when comparing many models the AIC is often inconclusive.
>> If one is bent on selecting just _the model_ then I check numerical
>> optimization diagnostics such as size of gradients, KKT criteria, and
>> other issues such as standard errors of parameter estimates and the
>> correlation matrix of parameter estimates.
> 
> -- And the mathematical basis for this claim is ....  ??
> 
> --
> Ruben: In my area of work (building/testing/applying mechanistic nonlinear
> models of natural and economic phenomena) the issue of numerical
> optimization is a very serious one. It is often the case that a really
> good looking model does not converge properly (that's why ADMB is so
> popular among us). So if the AIC is inconclusive, but one AIC-tied model
> yields reasonably looking standard errors and low pairwise parameter
> estimates correlations, while the other wasn´t even able to produce a
> positive definite Hessian matrix (though it was able to maximize the
> log-likelihood), I think I have good reasons to select the properly
> converged model. I assume here that the lack of convergence is a symptom
> of model inadequacy to the data, that the AIC was not able to detect.
> ---
> Ruben: For some reasons I don't find model averaging appealing. I guess
> deep in my heart I expect more from my model than just the best predictive
> ability.
> 
> -- This is a religious, not a scientific statement, and has no place in
> any scientific discussion.
> 
> --
> Ruben: Seriously, there is a wide and open place in scientific discussion
> for mechanistic model-building. I expect the models I built to be more
> than able predictors, I want them to capture some aspect of nature, to
> teach me something about nature, so I refrain from model averaging, which
> is an open admission that you don't care too much about what's really
> going on.
> 
> -- The belief that certain data analysis practices -- standard or not --
> somehow can be trusted to yield reliable scientific results in the face of
> clear theoretical (mathematical )and practical results to the contrary,
> while widespread, impedes and often thwarts the progress of science,
> Evidence-based medicine and clinical trials came about for a reason. I
> would encourage you to reexamine the basis of your scientific practice and
> the role that "magical thinking" plays in it.
> 
> Best,
> Bert
> 
> --
> Ruben: All right Bert. I often re-examine the basis of my scientific
> praxis but less often than I did before, I have to confess. I like to
> think it is because I am converging on the right praxis so there are less
> issues to re-examine. But this problem of model selection is a tough one.
> Being a likelihoodist in inference naturally leads you to AIC-based model
> selection, Jim Lindsey being influent too. Wanting that your models say
> some something about nature is another strong conditioning factor. Put
> this two together and it becomes hard to do model-averaging. And it has
> nothing to do with religion (yuck!).
> 
> ______________________________________________
> R-help@ mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 


-----
Frank Harrell
Department of Biostatistics, Vanderbilt University
--
View this message in context: http://r.789695.n4.nabble.com/How-do-I-compare-47-GLM-models-with-1-to-5-interactions-and-unique-combinations-tp4326407p4333464.html
Sent from the R help mailing list archive at Nabble.com.