Statistical Performance and Clinical Utility of Risk Prediction Models

• Activity Number: 442
• Type: Invited
• Date/Time: Wednesday, August 1, 2012 : 8:30 AM to 10:20 AM
• Sponsor: Biometrics Section
• Author(s): Nancy R. Cook*+
• Companies: BWH
• Keywords: prediction ; reclassification ; discrimination

Abstract

Risk reclassification methods have become popular in the medical literature as a means of comparing risk prediction models. Cross-stratification into clinical risk categories can be used to assess the potential impact of new models on treatment decisions for individual patients. Alternative continuous measures of model discrimination include the change in the area under the ROC curve (AUC), the integrated discrimination improvement (IDI), and the continuous net reclassification improvement (NRI(>0)). While theoretically interesting, these continuous markers may not be the most appropriate to assess clinical utility. The NRI(>0), in particular, is a measure of effect rather than model improvement and can sometimes exhibit erratic behavior. Caution is needed before using this as a measure of improvement. Further, the test of the NRI(>0), as well as that for the IDI, are similar to the likelihood ratio test in nested models, and may be over-interpreted. Reclassification in risk strata, while requiring thresholds, may be more relevant clinically with its ability to examine potential changes in treatment decisions.