Approximating Cross-Validation: Guarantees for Model Assessment and Selection

• Activity Number: 437 - Misspecification, Robustness, and Model Assessment
• Type: Topic-Contributed
• Date/Time: Thursday, August 12, 2021 : 4:00 PM to 5:50 PM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): Ashia Wilson*
• Companies: MIT

Abstract

Cross-validation (CV) is the de facto standard for selecting accurate predictive models and assessing model performance. However, CV suffers from a need to repeatedly refit a learning procedure on a large number of training datasets. To reduce the computational burden, a number of works have introduced approximate CV procedures that simultaneously reduce runtime and provide model assessments comparable to CV when the prediction problem is sufficiently smooth. An open question however is whether these procedures are suitable for model selection. In this talk, I’ll describe (i) broad conditions under which the model selection performance of approximate CV nearly matches that of CV, (ii) examples of prediction problems where approximate CV selection fails to mimic CV selection, and (iii) an extension of these results and the approximate CV framework more broadly to non-smooth prediction problems like L1-regularized empirical risk minimization.