A Significance Test in Forward Stepwise Model Selection

• Activity Number: 510
• Type: Contributed
• Date/Time: Wednesday, August 6, 2014 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Joshua Loftus*+ and Jonathan Taylor
• Companies: Stanford University and Stanford University
• Keywords: forward stepwise ; model selection ; p-value ; gaussian processes

Abstract

Recent work on hypothesis testing in regularized regression has yielded a new test statistic with an exact distribution under the global null only assuming Gaussian errors. Leaving the regularized regression setting, we iteratively apply this hypothesis test at each step in forward stepwise regression. While the model selection process invalidates traditional test statistics based on Chi-square and F-distributions, our method appears to be robust to selection and yields good inferences in a wide variety of settings. Instead of inference after selection, these p-values can also be used to form stopping rules for forward stepwise. Some simple stopping rules are explored via simulations and found to perform well in terms of average power and control of false discovery rates.