|
Abstract:
|
The change in AUC (delta AUC), the IDI and NRI are commonly used measures of predictive model performance. However, a risk prediction model enhanced with uninformative predictor(s) can lead to a significant improvement of NRI, and a model enhanced with a strong predictor can fail to produce a significant increase in AUC. Several explanations of these phenomena have been suggested, i.e. incorrect variance estimator, lack of training-validation approach and a flaw in the statistics itself. In this paper we unite the delta AUC, IDI and NRI under the umbrella of U-statistics family. Using powerful statistical theory developed for U-statistics, we explained several reported contradictions in their behavior. We proved that delta AUC, IDI and three types of NRI asymptotically follow normal distribution, unless they compare nested models under the null. In the latter case delta AUC, NRI and IDI are non-normal and most CI formulas are invalid. Using results of Randles, de Wet and Sukhatme we discuss, when their variance formulas should be adjusted for estimated parameters and when such adjustment is unnecessary.
|
