35 – Statistics in Sports, Competitions, and the Arts
Measuring Judge Agreement for a High School Diving Competition
Monnie McGee
Southern Methodist University
Jing Cao
Southern Methodist University
In a high school championship diving competition, athletes are asked to perform 11 dives. Each of 5 judges gives each dive a rating from 0, for a failed dive, to 10 for a perfect dive. Round scores are determined by dropping the lowest and highest judge's rating, adding the remaining three ratings, and multiplying that sum by the degree of difficulty of the dive. The total score is determined by the sum of the round scores. Agreement in the context of multiple raters has been well explored in the literature. However, measuring agreement in the context of multiple raters, with multiple scores per rater, and with covariates influencing the ratings, has been much less explored. Data from a regional high school diving competition has all of these components. We examine various measures of interrater agreement from the literature, and discuss the shortcomings of each in the context of this data set.