Adaptive Sequential Model Selection

• Activity Number: 682
• Type: Topic Contributed
• Date/Time: Thursday, August 4, 2016 : 10:30 AM to 12:20 PM
• Sponsor: ENAR
• Author(s): William Fithian* and Jonathan Taylor and Robert Tibshirani and Ryan Tibshirani
• Companies: University of California at Berkeley and Stanford University and Stanford University and Carnegie Mellon University
• Keywords: Selective inference ; Hypothesis testing ; Goodness-of-fit testing ; Machine learning

Abstract

Many model selection algorithms produce a path of fits specifying a sequence of increasingly complex models. Given such a sequence and the data used to produce them, we consider the problem of choosing the least complex model that is not falsified by the data. Extending the selected-model tests of Fithian et al. (2014), we construct p-values for each step in the path which account for the adaptive selection of the model path using the data. In the case of linear regression, we propose two specific tests, which improve on the power of the saturated-model test of Tibshirani et al. (2014), sometimes dramatically.

To select a model, we can feed our single-step p-values as inputs into sequential stopping rules such as those proposed by G'Sell et al. (2013) and Li and Barber (2015), achieving control of the familywise error rate or false discovery rate (FDR) as desired. We show that our proposed constructions yield independent p-values, as required by the FDR-controlling rules.