Risk Inflation of Sequential Testing

• Activity Number: 36
• Type: Contributed
• Date/Time: Sunday, August 4, 2013 : 2:00 PM to 3:50 PM
• Sponsor: IMS
• Author(s): Robert Stine*+
• Companies: University of Pennsylvania
• Keywords: streaming feature selection ; data mining ; false discovery rate

Abstract

Sequential testing is an important component of streaming methods for very fast feature selection. This research finds the risk inflation of the estimator defined by a sequence of tests as used in streaming feature selection. The risk inflation of a sequence of tests is the worst-case ratio of the cumulative risk of the implied keep-or-kill estimator to the risk obtained by an oracle. Our analysis provides the exact risk inflation of alpha investing in a finite sequence of tests. These results show that the asymptotic bound 2log p from Foster and George (1994) holds for most alpha investing procedures in the context of finite sequential testing. Alpha investing produces low risk when the procedure is allowed to produce a surprisingly large number of false rejections, and the traditional choice of a 5% error rate produces substantial risk inflation in the presence of moderate amounts of signal. Our results also characterize stochastic processes that generate parameters that produces the maximum risk inflation.