Computationally Efficient Confidence Intervals for Cross-Validated AUC Estimates

• Activity Number: 611
• Type: Contributed
• Date/Time: Wednesday, August 7, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Erin LeDell*+ and Maya Petersen and Mark Van der Laan
• Companies: UC Berkeley and UC Berkeley - Biostatistics and UC Berkeley - Biostatistics
• Keywords: Variance estimation ; Cross-validation ; Confidence intervals ; AUC ; Performance evaluation ; Bootstrap alternative

Abstract

In binary classification problems, cross-validated area under the ROC curve (AUC) is a popular means of measuring the performance of your model. In order to evaluate the quality of an estimate for cross-validated AUC, we must obtain an estimate for its variance. When using ensemble learners or large data sets, the time required to generate the cross-validated AUC estimate can be significant. In these cases, the bootstrap, as a means of variance estimation, can be computationally intractable. As an alternative to the bootstrap, we demonstrate a computationally efficient influence curve based approach to obtaining a variance estimate for cross-validated AUC.

These methods for computing confidence intervals for cross-validated AUC estimates for both i.i.d. and pooled repeated measures data have been implemented in an R package called cvAUC, available on CRAN.