Simpson's Paradox in the Integrated Discrimination Improvement

• Activity Number: 173
• Type: Topic Contributed
• Date/Time: Monday, August 10, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Section on Risk Analysis
• Author(s): Jonathan Chipman* and Danielle Braun
• Companies: Vanderbilt University and Harvard School of Public Health
• Keywords: Integrated Discrimination Improvement ; Simpson's Paradox ; Risk Prediction ; Reclassification ; BRCAPRO

Abstract

The Integrated Discrimination Improvement (IDI) is a commonly used metric to compare two risk prediction models; it summarizes the extent to which a new model increases risk in events and decreases risk in non-events. The IDI averages risks across events and non-events, and is therefore susceptible to Simpson's Paradox. In some settings, adding a predictive covariate to a well calibrated model results in an overall negative (positive) IDI. However if we stratify by that same covariate, the strata-specific IDIs are positive (negative). Meanwhile, the calibration (observed to expected ratio, O/E), area under the receiver-operating-characteristic curve (AUC), and Brier score improve overall and for each stratum. We ran extensive simulations to investigate the impact of an imbalanced covariate upon metrics (IDI, AUC, Brier score, and $R^2$), provide an analytic explanation for the paradox in the IDI, and explore a simple modification to be used in these settings. The paradox is illustrated on data from the Cancer Genomics Network, by calculating predictions based on two versions (v2.0-8 and v2.0-7) of BRCAPRO, a Mendelian risk prediction model for breast and ovarian cancer.