[R] predictive accuracy

[Marc Schwartz]

On May 26, 2011, at 7:42 AM, El-Tahtawy, Ahmed wrote:

> I am trying to develop a prognostic model using logistic regression.   I
> built a full , approximate models with the use of penalization - design
> package. Also, I tried Chi-square criteria, step-down techniques. Used
> BS for model validation. 
> 
> 
> 
> The main purpose is to develop a predictive model for future patient
> population.   One of the strong predictor pertains to the study design
> and would not mean much for a clinician/investigator in real clinical
> situation and have been asked to remove it.
> 
> 
> 
> Can I propose a model and nomogram without that strong -irrelevant
> predictor?? If yes, do I need to redo model calibration, discrimination,
> validation, etc...?? or just have 5 predictors instead of 6 in the
> prognostic model??
> 
> 
> 
> Thanks for your help
> 
> Al


Is it that the study design characteristic would not make sense to a clinician but is relevant to future samples, or that the study design characteristic is unique to the sample upon which the model was developed and is not relevant to future samples because they will not be in the same or a similar study?

Is the study design characteristic a surrogate for other factors that would be relevant to future samples? If so, you might engage in a conversation with the clinicians to gain some insights into other variables to consider for inclusion in the model, that might in turn, help to explain the effect of the study design variable.

Either way, if the covariate is removed, you of course need to engage in fully re-evaluating the model. You cannot just drop the covariate and continue to use model fit assessments made on the full model.

HTH,

Marc Schwartz