Sequential Changepoint Detection for Classifier Label Shift

• Activity Number: 498 - Modern Machine Learning
• Type: Contributed
• Date/Time: Thursday, August 6, 2020 : 10:00 AM to 2:00 PM
• Sponsor: Section on Statistical Learning and Data Science
• Author(s): Ciaran Evans* and Max G'Sell
• Companies: Carnegie Mellon University and Carnegie Mellon University
• Keywords: changepoint detection; classification; label shift; sequential analysis; density estimation

Abstract

When applying a classifier to new data, shifts in the overall base rate of labels in the population -- known as label shift -- can lead to miscalibration of the classifier scores and significant decreases in performance. Furthermore, a shift in the overall base rate is often of practical interest in its own right. In this work, we develop a nonparametric sequential changepoint detection procedure for rapidly detecting such shifts in the population base rate based on classifier scores, without requiring observation labels. We provide methods for approximating the operating characteristics of our proposal, and show that performance can be comparable to the optimal detection procedure. In the course of this theoretical development, we also develop more general results for nonparametric changepoint detection.