Online Program Home
  My Program

All Times EDT

Abstract Details

Activity Number: 225 - Contributed Poster Presentations: Section on Medical Devices and Diagnostics
Type: Contributed
Date/Time: Tuesday, August 4, 2020 : 10:00 AM to 2:00 PM
Sponsor: Section on Medical Devices and Diagnostics
Abstract #313454
Title: Novel Insights for Measuring Interrater Agreement: Exploring Fundamental Principles
Author(s): Katelyn McKenzie* and Jonathan Mahnken
Companies: University of Kansas Medical Center and Department of Biostatistics and Data Science, The University of Kansas Medical Center
Keywords: interrater agreement; multinomial probabilities; Cohen's kappa; marginal probabilities

Often, agreement studies utilize limited raters, and frequently only two. Therefore, understanding the underlying mechanisms that contribute to interrater agreement are important to the effective use and appropriate interpretation of traditional methods. One approach for studying these mechanisms is by examining interrater agreement in the context of diagnostic interpretation by true disease status. Notably, Cohen’s kappa can be rewritten to reveal insights as to how kappa relates to other quantities, such as the odds of observed agreement under independence. Hence, this re-expression of kappa yields insight into how the joint distribution of diagnostic interpretation by true disease status affects the marginal probability of each rating. Further, samples typically selected for diagnostic testing may not provide an adequate representation of the population of interest, which may affect inferential assumptions. We show how kappa can be rewritten, examine different sampling paradigms and study the advantages of competing designs. These new, yet fundamental, insights provide a different lens with which to study and interpret interrater agreement.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2020 program