Online Program Home
  My Program

Abstract Details

Activity Number: 412 - Applications and Methods for Risk Estimation
Type: Contributed
Date/Time: Tuesday, August 1, 2017 : 2:00 PM to 3:50 PM
Sponsor: Section on Risk Analysis
Abstract #324449
Title: Measures of Predictive Model Performance and Event Rate: How Much They Vary and How to Make Them Comparable Across Studies
Author(s): Olga Demler* and Nancy R. Cook
Companies: Harvard Medical School and Harvard T. H. Chan School of Public Health
Keywords: risk prediction ; AUC ; NRI ; IDI ; Brier Score
Abstract:

The performance of risk prediction models can be evaluated in terms of discrimination, calibration and reclassification using such metrics as delta AUC, NRI, IDI, Brier Score, Net Benefit and other measures. Some of these metrics are invariant with respect to the underlying event rate (such as the AUC) while others vary with the event rate. We explore the association of these measures of model performance with the event rate. We show which metrics are invariant and which are not. We find that some non-invariant metrics can stay relatively flat over a wide range of event rates while others can vary substantially by event rate. It is thus difficult to compare such metrics across studies. For example, a lower IDI in one study may be explained by a lower event rate and not by weaker model improvement. We discuss the implications of this and provide practical recommendations on how to make these measures comparable across studies.


Authors who are presenting talks have a * after their name.

Back to the full JSM 2017 program

 
 
Copyright © American Statistical Association