A Risk Stratification Approach for Improved Interpretation of Diagnostic Accuracy Statistics

• Activity Number: 216 - Statistical Opportunities in Disease Interception: Screening, Intervening, and Evaluating Benefit-Risk Trade-Offs
• Type: Topic Contributed
• Date/Time: Monday, July 31, 2017 : 2:00 PM to 3:50 PM
• Sponsor: Section on Medical Devices and Diagnostics
• Author(s): Hormuzd Katki* and Mark Schiffman
• Companies: National Cancer Institute and National Cancer Institute
• Keywords: risk prediction ; AUC ; diagnostic testing

Abstract

We interpret standard diagnostic accuracy statistics, such as Youden index and AUC, in light of risk-stratification (how well a biomarker separates those at higher risk from those at lower risk) to better understand implications for public-health programs. We introduce an intuitive statistic, Mean Risk Stratification (MRS): the average change in risk (pre-test vs. post-test) revealed for tested individuals. MRS is a function of both AUC and disease prevalence, and thus little risk-stratification is possible for rare diseases (regardless of AUC), demonstrating a high-bar to justify population-based screening. AUC measures multiplicative relative gains in risk-stratification: AUC=0.6 achieves only 20% of maximum risk-stratification (AUC=0.9 achieves 80%). However, large relative gains in risk-stratification might not imply large absolute gains if disease is rare. We use MRS to compare cervical cancer screening tests in China vs. USA. The test with the worst AUC=0.72 in China (visual inspection with ascetic acid) provides twice the MRS of the test with best AUC=0.83 in the USA (HPV and Pap cotesting) because China has three times more cervical precancer/cancer.