Challenges with the Use of Cross-Validation for Comparing Structured Models

• Activity Number: 541
• Type: Contributed
• Date/Time: Wednesday, August 7, 2013 : 10:30 AM to 12:20 PM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): Wei Wang*+ and Andrew Gelman
• Companies: and Columbia University
• Keywords: model comparison ; cross-validation ; Bayesian hierarchical models ; survey methodology

Abstract

As a simple and compelling approach for estimating out-of-sample predictive error, cross-validation naturally lends itself to the task of model comparison. However, we feel that the legitimacy of cross-validation methods in model comparison is often being taken for granted. In this work, we want to clarify what cross-validation methods are measuring when they are used for model comparison. Using a hierarchical model fit to large survey data with a battery of questions, we show that even though cross-validation might give good estimates of out-of-sample performance, it is not always a sensitive instrument for model comparison. In addition, we emphasize the importance of proper calibration in assisting us interpret the practical importance of the results when conducting model comparison.