Evaluation and Selection of Models for Out-of-Sample Prediction When the Sample Size Is Small Relative to the Complexity of the Data-Generating Process

• Activity Number: 544
• Type: Contributed
• Date/Time: Thursday, August 2, 2007 : 10:30 AM to 12:20 PM
• Sponsor: Section on Physical and Engineering Sciences
• Author(s): Hannes Leeb*+
• Companies: Yale University
• Keywords: model selection ; out-of-sample prediction ; generalized cross validation ; S_p criterion ; non-parametric regression ; large number of parameters and small sample size

Abstract

We study the problem of selecting a model that performs well for out-of-sample prediction. We do not assume that any of the candidate models under consideration is correct. Our analysis is based on explicit finite-sample results. The results are non-standard because we consider a situation where the sample size is small relative to the complexity of the data-generating process. Also, we allow for the case where the number of candidate models is (much) larger than sample size. For Gaussian data, we show under minimal assumptions that model selection based on generalized cross validation or the S_p criterion performs well in such situations, uniformly over large regions in parameter space. We also show that the performance of other model selectors, including AIC and BIC, can be anything from satisfactory or mildly suboptimal to completely unreasonable, depending on unknown parameters.