Limit Theory for Comparing Overfit Models Out-of-Sample

• Activity Number: 298
• Type: Contributed
• Date/Time: Tuesday, August 3, 2010 : 8:30 AM to 10:20 AM
• Sponsor: Business and Economic Statistics Section
• Author(s): Gray Calhoun*+
• Companies: Iowa State University
• Keywords: forecasting ; dimension asymptotics ; forecast evaluation ; model comparison ; hypothesis test

Abstract

This paper studies the rationale for comparing overfit forecasts out-of-sample instead of in-sample. The two models to be compared are nested linear regressions and the number of predictors used by the larger model, k, increases with the number of observations, T , so that the ratio k/T remains uniformly positive. Under this limit theory, tests that are designed to reject if the larger model is true, such as the usual in-sample Wald and LM test and also popular out-of-sample statistics, will choose the larger model too often when the smaller model is more accurate. We show that the usual out-of-sample test using Gaussian critical values is valid in the following sense: it will incorrectly choose the larger model with probability equal to the test's nominal size. We also show that the out-of-sample period must be small relative to the sample size when the models are overfit.