Online Program Home
My Program

Abstract Details

Activity Number: 493
Type: Contributed
Date/Time: Wednesday, August 3, 2016 : 8:30 AM to 10:20 AM
Sponsor: Section on Risk Analysis
Abstract #320073
Title: Asymptotic Distribution of Delta AUC, NRI, and IDI Based on U-Statistics Theory
Author(s): Olga Demler* and Michael Pencina and Ralph D'Agostino
Companies: Harvard Medical School and Duke University and Boston University
Keywords: AUC ; NRI ; IDI ; U-statistics ; risk prediction ; discrimination

The change in AUC (delta AUC), the IDI and NRI are commonly used measures of predictive model performance. However, a risk prediction model enhanced with uninformative predictor(s) can lead to a significant improvement of NRI, and a model enhanced with a strong predictor can fail to produce a significant increase in AUC. Several explanations of these phenomena have been suggested, i.e. incorrect variance estimator, lack of training-validation approach and a flaw in the statistics itself. In this paper we unite the delta AUC, IDI and NRI under the umbrella of U-statistics family. Using powerful statistical theory developed for U-statistics, we explained several reported contradictions in their behavior. We proved that delta AUC, IDI and three types of NRI asymptotically follow normal distribution, unless they compare nested models under the null. In the latter case delta AUC, NRI and IDI are non-normal and most CI formulas are invalid. Using results of Randles, de Wet and Sukhatme we discuss, when their variance formulas should be adjusted for estimated parameters and when such adjustment is unnecessary.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2016 program

Copyright © American Statistical Association