Using Stochastic Orders to Evaluate Performance of Predictive Models: Intransitivity, Area Under the ROC, and Strength of Stochastic Order Relationships

• Activity Number: 233 - Evaluating Individualized Predictions of Risk and Benefit for Clinical Use
• Type: Topic-Contributed
• Date/Time: Wednesday, August 11, 2021 : 10:00 AM to 11:50 AM
• Sponsor: WNAR
• Author(s): Olga Demler*
• Companies: Harvard University

Abstract

Area Under the Receiver Operating Characteristics Curve (AUC of ROC also known as a c-statistics) is a commonly used to evaluate discriminatory ability of two-class classification models. AUC is used to evaluate a wide range of models from Neural Networks to logistic regression. AUC cannot be used for evaluating 3-class and higher ordinal classification models because AUC-based comparisons can be intransitive. Intransitivity is best explained using intransitive (or Efron) dice. Dice A, B and C are called intransitive when they use such a layout of scores that die A beats more often die B, die B beats more often die C, die C paradoxically beats more often die A. We provide examples of intransitive relationships from real life using chess and other sports tournaments and network meta-analysis. We then extend Efron dice (discreet binomial case) to continuous distribution functions. We then argue that intransitivity is a sign of weakness of discriminatory ability of a model, yet nominal value of AUC and its confidence intervals can mask this weakness, therefore we need a model performance measure that will alert us to this weakness. One way to address it is to use Trybula-Savage limit