Measuring the Performance Of Medical Screening Tests: Average Precision vs. Area Under ROC Curve

• Activity Number: 265
• Type: Contributed
• Date/Time: Monday, August 5, 2013 : 2:00 PM to 3:50 PM
• Sponsor: SSC
• Author(s): Yan Yuan*+ and Wanhua Su and Mu Zhu
• Companies: University of Alberta and MacEwen University and University of Waterloo
• Keywords: Average Precision ; Receiver Operating Characteristic (ROC) curve ; AUC (Area under the ROC curve) ; Asymptotic Variance ; Medical screening

Abstract

Analogous to the receiver operating characteristic (ROC) curve, the hit curve is often used to evaluate binary classification algorithms, especially in information retrieval. The average precision (AP) is a numeric summary of the hit curve, much like the area under the ROC curve (AUC). However, the AP is sometimes viewed as being ad hoc, which has limited its use by the wider scientific community. In this talk, we first show that both the AP and the AUC can be calculated as a weighted sum of precisions along the hit curve, and that both their weights are gradually decreasing, except those for the AP are data-adaptive whereas those for the AUC are not. This makes the AP attractive, especially if the earlier part of the ROC curve (or the hit curve) is of interest. Then, we derive the asymptotic variance for the AP and verify its accuracy with simulated data. Finally, we compare the two performance measures using data set from the medical field.