In Practice, Many Models Fit the Data Equally Well

• Activity Number: 249
• Type: Contributed
• Date/Time: Monday, July 30, 2012 : 2:00 PM to 3:50 PM
• Sponsor: WNAR
• Author(s): John Boscardin*+
• Companies: University of California at San Francisco
• Keywords: score selection ; automated selection ; leaps and bounds

Abstract

Model selection often focuses on identification of an optimal model according to some criterion measure. However, in many settings, a huge number of models turn out to be statistically indistinguishable, and searching for a single best model should probably not be the primary goal of model selection. I will discuss several settings for this principle including predictor selection in prognostic model building, choosing a model for the mean structure of longitudinal data, and identifying heterogeneous subgroups via latent trajectory analysis. In the predictor selection setting, the over-optimism of predictive accuracy measures, due to both (i) the process of predictor selection and to (ii) using estimates of the regression coefficients optimized to the data set, can be examined using bootstrapping. I will present an example comparing these two components of over-optimism for model building using best subset regression, using stepwise selection, and with no trimming of predictors. In the two settings with longitudinal data, I will show specific examples where a number of models fit very similarly but have importantly different substantive interpretations.