Cross-Validation Confidence Intervals for Test Error

• Activity Number: 63 - Inference and Interpretability in a Model-Free Setting
• Type: Invited
• Date/Time: Monday, August 9, 2021 : 10:00 AM to 11:50 AM
• Sponsor: IMS
• Author(s): Lester Mackey* and Pierre Bayle and Alexandre Bayle and Lucas Janson
• Companies: Microsoft Research New England and Princeton University and Harvard University and Harvard University
• Keywords: Cross-validation; Central limit theorem; Asymptotic variance; Algorithmic stability; Confidence intervals

Abstract

This work develops central limit theorems for cross-validation and consistent estimators of its asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our real-data experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.