Comparison of Statistical Methods in Assessing Predictive Models

• Activity Number: 68
• Type: Contributed
• Date/Time: Sunday, August 3, 2014 : 4:00 PM to 5:50 PM
• Sponsor: ENAR
• Author(s): Hui Zhou*+ and Jeffrey Slezak and Stephen F. Derose and Anny H. Xiang
• Companies: Kaiser Permanente and Kaiser Permanente and Kaiser Permanente and Kaiser Permanente
• Keywords: Predictive model ; calibration ; discrimination ; AUC ; net reclassification index ; integrated discrimination improvement

Abstract

Discrimination and calibration are two important aspects in evaluating predictive model performance. Multiple statistical methods such as area under the ROC curve (AUC), Hosmer-Lemeshow statistics, net reclassification index (NRI) and integrated discrimination improvement (IDI) have been proposed. Inconsistent conclusions often appeared when using these methods and debate exists for which method is superior. In this study, we assessed the above methods in evaluating the performance of various predictive models, including nested and independent models. Data were generated by a semi-simulated approach where Framingham predicted risk was applied to determine outcome. The outcome rate was assumed to be 3%, 14%, and 60% respectively. Our results showed that in ranking the performance of the various models, AUC, NRI, and IDI always reached same conclusions, however, Hosmer-Lemeshow statistics often provided different conclusions. We conclude that AUC, NRI, and IDI generally will provide similar results when common risk thresholds are used, since all measure model discrimination. However, Hosmer-Lemeshow statistics should be reported separately for the purpose of model calibration.