Neyman-Pearson Classification Algorithms and NP Receiver Operating Characteristics

• Activity Number: 1 - Invited E-Poster Session
• Type: Invited
• Date/Time: Sunday, August 2, 2020 : 12:30 PM to 3:30 PM
• Sponsor: WNAR
• Author(s): Jingyi Jessica Li* and Xin Tong and Yang Feng
• Companies: University of California, Los Angeles and University of Southern California and New York University
• Keywords: binary classification; asymmetric errors; high probability bound; machine learning; Neyman-Pearson; ROC

Abstract

In many binary classification applications, such as disease diagnosis and spam detection, practitioners commonly face the need to limit type I error (that is, the conditional probability of misclassifying a class 0 observation as class 1) so that it remains below a desired threshold. To address this need, the Neyman-Pearson (NP) classification paradigm is a natural choice; it minimizes type II error (that is, the conditional probability of misclassifying a class 1 observation as class 0) while enforcing an upper bound, alpha, on the type I error. Common practices that directly control the empirical type I error to alpha lead to classifiers that are likely to have population type I errors much larger than alpha. We develop the first umbrella algorithm that implements the NP paradigm for all scoring-type classification methods, such as logistic regression, support vector machines, and random forests. Powered by this algorithm, we developed the NP receiver operating characteristic (NP-ROC) bands motivated by the popular ROC curves. NP-ROC bands will help choose alpha in a data-adaptive way and compare classification methods under the NP paradigm.