Evaluating Calibration of Risk Prediction Models

• Activity Number: 520
• Type: Topic Contributed
• Date/Time: Wednesday, August 12, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Section on Risk Analysis
• Author(s): Ruth Pfeiffer*
• Companies: National Cancer Institute
• Keywords: absolute risk ; calibration ; model validation ; prediction ; bias

Abstract

Statistical models that predict disease incidence, disease recurrence or mortality following disease onset have broad public health and clinical applications. Before a model can be recommended for practical use, its performance characteristics need to be understood. General criteria to evaluate prediction models for dichotomous outcomes include predictive accuracy, proportion of variation explained, calibration and discrimination. Most recent validation studies have emphasized calibration and discrimination. A model is called "well calibrated" (or unbiased) when the predicted probabilities agree with observed risk in subsets of the population and overall. I propose and study novel criteria to assess the calibration of models that predict risk of disease incidence and compare their performance to standard methods to assess model calibration. I illustrate the methods with models that predict incidence of endometrial and breast cancer.