Online Program Home
  My Program

All Times EDT

Abstract Details

Activity Number: 233 - Evaluating Individualized Predictions of Risk and Benefit for Clinical Use
Type: Topic-Contributed
Date/Time: Wednesday, August 11, 2021 : 10:00 AM to 11:50 AM
Sponsor: WNAR
Abstract #317652
Title: Using Stochastic Orders to Evaluate Performance of Predictive Models: Intransitivity, Area Under the ROC, and Strength of Stochastic Order Relationships
Author(s): Olga Demler*
Companies: Harvard University
Keywords:
Abstract:

Area Under the Receiver Operating Characteristics Curve (AUC of ROC also known as a c-statistics) is a commonly used to evaluate discriminatory ability of two-class classification models. AUC is used to evaluate a wide range of models from Neural Networks to logistic regression. AUC cannot be used for evaluating 3-class and higher ordinal classification models because AUC-based comparisons can be intransitive. Intransitivity is best explained using intransitive (or Efron) dice. Dice A, B and C are called intransitive when they use such a layout of scores that die A beats more often die B, die B beats more often die C, die C paradoxically beats more often die A. We provide examples of intransitive relationships from real life using chess and other sports tournaments and network meta-analysis. We then extend Efron dice (discreet binomial case) to continuous distribution functions. We then argue that intransitivity is a sign of weakness of discriminatory ability of a model, yet nominal value of AUC and its confidence intervals can mask this weakness, therefore we need a model performance measure that will alert us to this weakness. One way to address it is to use Trybula-Savage limit


Authors who are presenting talks have a * after their name.

Back to the full JSM 2021 program