[R] Bayesian regression stepwise function?

Frank E Harrell Jr f.harrell at vanderbilt.edu
Fri Oct 23 23:21:13 CEST 2009


Ravi Varadhan wrote:
> Frank,
> 
> I have heard this (i.e. only head-to-head comparisons are valid) and various
> other folklores about AIC and BIC based model selection before, including
> one that these information criteria are only applicable for comparing two
> nested models.  
> 
> Where has it been demonstrated that AIC/BIC cannot be used to find the best
> subset, i.e. the subset that is closest to the "true" model (assuming that
> true model is contained in the set of models considered, and that maximum
> likelihood estimation is used for estimating parameters in the models)?  
> 
> I would greatly appreciate any reference that shows this.
> 
> Best,
> Ravi. 

I don't have a specific reference, but the AIC/BIC approach, when 
entertaining more than a handful of models, has a very low probability 
of selecting the 'right' model.  AIC/BIC are just restatements of 
P-values (especially AIC), so they have all the many problems that 
P-values have.  It is for these reasons that Bayesians have much more 
success with model averaging than with model selection.  And if all the 
models averaged are within one class of models, it is more logical to 
use penalization, entertaining only one comprehensive model.  Penalized 
regression performs just as well as the much more complex model 
averaging approach if staying within one model class, and penalized 
models are easier to explain.

Frank

> 
> ----------------------------------------------------------------------------
> -------
> 
> Ravi Varadhan, Ph.D.
> 
> Assistant Professor, The Center on Aging and Health
> 
> Division of Geriatric Medicine and Gerontology 
> 
> Johns Hopkins University
> 
> Ph: (410) 502-2619
> 
> Fax: (410) 614-9625
> 
> Email: rvaradhan at jhmi.edu
> 
> Webpage:
> http://www.jhsph.edu/agingandhealth/People/Faculty_personal_pages/Varadhan.h
> tml
> 
>  
> 
> ----------------------------------------------------------------------------
> --------
> 
> 
> -----Original Message-----
> From: Frank E Harrell Jr [mailto:f.harrell at vanderbilt.edu] 
> Sent: Friday, October 23, 2009 4:04 PM
> To: Ravi Varadhan
> Cc: JLucke at ria.buffalo.edu; 'Allan.Y'; r-help at r-project.org;
> r-help-bounces at r-project.org
> Subject: Re: [R] Bayesian regression stepwise function?
> 
> Ravi Varadhan wrote:
>> The stepAIC() function in "MASS" can be used, with k = log(n), to
> implement
>> your suggestion of "quasi-Bayesian" stepwise selection using the BIC
>> criterion.
>>
>> Ravi.
> 
> Although many statisticians use BIC otherwise, it was only designed to 
> compare two pre-specified models.
> 
> Frank
> 
>>
> ----------------------------------------------------------------------------
>> -------
>>
>> Ravi Varadhan, Ph.D.
>>
>> Assistant Professor, The Center on Aging and Health
>>
>> Division of Geriatric Medicine and Gerontology 
>>
>> Johns Hopkins University
>>
>> Ph: (410) 502-2619
>>
>> Fax: (410) 614-9625
>>
>> Email: rvaradhan at jhmi.edu
>>
>> Webpage:
>>
> http://www.jhsph.edu/agingandhealth/People/Faculty_personal_pages/Varadhan.h
>> tml
>>
>>  
>>
>>
> ----------------------------------------------------------------------------
>> --------
>>
>>
>> -----Original Message-----
>> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
> On
>> Behalf Of JLucke at ria.buffalo.edu
>> Sent: Friday, October 23, 2009 2:31 PM
>> To: Allan.Y
>> Cc: r-help at r-project.org; r-help-bounces at r-project.org
>> Subject: Re: [R] Bayesian regression stepwise function?
>>
>> The BIC (Raftery) can be used for quasi-Bayesian model selection, but it's
> 
>> not stepwise.   Ntzoufras shows how to use WinBUGS to conduct Bayesian 
>> model selection, but again it's not stepwise
>>
>>
>> Ntzoufras, I. (2002), 'Gibbs variable selection using BUGS', Journal of 
>> Statistical Software 7(7), 1--19.
>> Ntzoufras, I. (2009), Bayesian modeling using WinBUGS, Wiley, Hoboken, NJ.
>> Raftery, A. E. (1995), 'Bayesian model selection in social research', 
>> Sociological Methodology 25, 111-163.
>>
>>
>>
>>
>>
>>
>>
>> "Allan.Y" <allany at cmu.edu> 
>> Sent by: r-help-bounces at r-project.org
>> 10/22/2009 01:09 PM
>>
>> To
>> r-help at r-project.org
>> cc
>>
>> Subject
>> [R]  Bayesian regression stepwise function?
>>
>>
>>
>>
>>
>>
>>
>> Hi everyone,
>>
>> I am wondering if there exists a stepwise regression function for the
>> Bayesian regression model.  I tried googling, but I couldn't find 
>> anything. 
>> I know "step" function exists for regular stepwise regression, but nothing
>> for Bayes.
>>
>>
>> Thanks
> 
> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University




More information about the R-help mailing list