Misuse of Delong Test to Compare AUCs for Nested Models

• Activity Number: 407
• Type: Contributed
• Date/Time: Tuesday, August 2, 2011 : 2:00 PM to 3:50 PM
• Sponsor: Biometrics Section
• Author(s): Olga Demler*+ and Michael Pencina and Ralph B. D'Agostino Sr.
• Companies: Boston University and Boston University and Boston University
• Keywords: AUC ; ROC ; risk prediction ; DeLong test ; logistic regression ; model discrimination

Abstract

Area under the Receiver Operating Characteristics Curve, (AUC of ROC) is a widely used measure of discrimination in risk prediction models. Mann-Whitney statistics is used as a non-parametric estimator of AUC. The difference of two AUCs is often tested by DeLong test. This study was motivated by numerous reports that often the added predictor is statistically significantly associated with the outcome but fails to produce significant improvement in the AUC. We suggest a possible explanation. We show that DeLong test can not be applied to test AUC improvement for nested models for any continuous distribution of the data and very general class of statistical models including logistic regression. First we show empirically that distribution of the difference of two AUCs from nested models is very different from the one used by the DeLong test. We use theory of U-statistics to explain this contradiction by showing that the difference of two AUCs belongs to a degenerate class of U-statistics and therefore has different asymptotic distribution than the one used by the DeLong test. It results in substantial (up to 60%) loss of power by the DeLong test. Possible solutions are discussed.