Skill Curves and ROC Curves for Diagnoses, or Why Skill Curves are More Fun

• Activity Number: 310
• Type: Topic Contributed
• Date/Time: Tuesday, August 9, 2005 : 2:00 PM to 3:50 PM
• Sponsor: Biometrics Section
• Author(s): William Briggs*+ and Russell Zaretzki and David Ruppert
• Companies: Cornell University and University of Tennessee and Cornell University
• Keywords: skill ; forecast ; roc curve ; prediction ; classification

Abstract

The skill score is a measure that compares expert and optimal naive forecasts for dichotomous predictions. We define as skillful a set of expert predictions with smaller expected loss than a set of optimal naive predictions. The term optimal naïve predictions refers to predictions made knowing only the marginal probability of the event, P(Y=1), and the loss for making incorrect predictions. Forecasts without skill should not be used. Skill also can be computed for diagnostic situations where a measurement X is used to predict an underlying condition, such as disease. For example, if white blood cell count exceeds threshold h, the diagnosis (prediction) is for Y = 1 (appendicitis present). We demonstrate the effectiveness of plotting skill for differing cutoff values and compare these to ROC curves. We show that skill curves are easier to interpret than ROC curves. Further, h* that maximizes skill is shown to give the optimal Bayes classification rule. This is equivalent to the optimal decision threshold for ROC curve analysis.