Optimal Streaming Feature Selection

• Activity Number: 249
• Type: Contributed
• Date/Time: Monday, July 30, 2012 : 2:00 PM to 3:50 PM
• Sponsor: IMS
• Author(s): Robert Stine*+
• Companies: University of Pennsylvania
• Keywords: model selection ; alpha investing ; multiple testing

Abstract

Streaming feature selection identifies variables to add to a model by testing a sequence of proposed variables. Rather than specify a set of explanatory variables as in the typical stepwise regression, streaming selection allows the search for predictive variables to adapt to the success of prior choices. The choice of the next variable can depend on results of prior tests. To avoid over-fitting, alpha-investing controls the false discovery rate of these sequential tests. Alpha-investing rules, however, provide the modeler with considerable flexibility in both the selection of features to test as well as the alpha-level committed to each test. Our work here identifies optimal strategies for alpha-investing that perform within a tolerance of the best possible. The modeler is then free to focus on strategies for choosing the next explanatory variable rather than setting the level of the next test. We show that a spending rule based on a universal prior is competitive with an oracle that knows the strategy of the modeler and sets the signal of the tested hypothesis. We also consider the risk-ratio of the selected model to that of the oracle model.