On Data Splitting for Model Validation

• Activity Number: 330
• Type: Topic Contributed
• Date/Time: Tuesday, July 31, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Section on Nonparametric Statistics
• Author(s): Serge Prudhomme*+ and Rebecca Morrison and Corey Bryant and Gabriel Terejanu and Kenji Miki
• Companies: The University of Texas at Austin and The University of Texas at Austin and The University of Texas at Austin and The University of Texas at Austin and The University of Texas at Austin
• Keywords: model validation ; predictive modeling ; cross-validation ; mathematical models

Abstract

Model validation implicitly assumes that datasets be split into calibration and validation sets, but how to do so is often subjective. We present an algorithm inspired from cross-validation as an attempt to provide a systematic solution to this problem. By considering several possible splits, we post-process the results to determine an optimal partition of the data subject to given constraints. We are interested here in the prediction of quantities of interest using mathematical models of physical systems for making critical decisions. The proposed approach addresses two critical issues: 1) the model should be assessed with respect to its ability to reproduce the data and to provide reliable predictions of a quantity of interest and 2) the model should be highly challenged by the validation set. The proposed framework is fairly general and may be applied to a wide range of problems. In this presentation, we will illustrate on several examples the main features of the framework.