Abstract Details
Activity Number:
|
68
|
Type:
|
Contributed
|
Date/Time:
|
Sunday, August 3, 2014 : 4:00 PM to 5:50 PM
|
Sponsor:
|
ENAR
|
Abstract #311339
|
|
Title:
|
Comparison of Statistical Methods in Assessing Predictive Models
|
Author(s):
|
Hui Zhou*+ and Jeffrey Slezak and Stephen F. Derose and Anny H. Xiang
|
Companies:
|
Kaiser Permanente and Kaiser Permanente and Kaiser Permanente and Kaiser Permanente
|
Keywords:
|
Predictive model ;
calibration ;
discrimination ;
AUC ;
net reclassification index ;
integrated discrimination improvement
|
Abstract:
|
Discrimination and calibration are two important aspects in evaluating predictive model performance. Multiple statistical methods such as area under the ROC curve (AUC), Hosmer-Lemeshow statistics, net reclassification index (NRI) and integrated discrimination improvement (IDI) have been proposed. Inconsistent conclusions often appeared when using these methods and debate exists for which method is superior. In this study, we assessed the above methods in evaluating the performance of various predictive models, including nested and independent models. Data were generated by a semi-simulated approach where Framingham predicted risk was applied to determine outcome. The outcome rate was assumed to be 3%, 14%, and 60% respectively. Our results showed that in ranking the performance of the various models, AUC, NRI, and IDI always reached same conclusions, however, Hosmer-Lemeshow statistics often provided different conclusions. We conclude that AUC, NRI, and IDI generally will provide similar results when common risk thresholds are used, since all measure model discrimination. However, Hosmer-Lemeshow statistics should be reported separately for the purpose of model calibration.
|
Authors who are presenting talks have a * after their name.
Back to the full JSM 2014 program
|
2014 JSM Online Program Home
For information, contact jsm@amstat.org or phone (888) 231-3473.
If you have questions about the Professional Development program, please contact the Education Department.
The views expressed here are those of the individual authors and not necessarily those of the JSM sponsors, their officers, or their staff.
Copyright © American Statistical Association.