Online Program Home
  My Program

All Times EDT

Abstract Details

Activity Number: 346 - Advances in Diagnostics and Reproducibility Research
Type: Topic-Contributed
Date/Time: Thursday, August 12, 2021 : 10:00 AM to 11:50 AM
Sponsor: Social Statistics Section
Abstract #317247
Title: A Paired Kappa for Comparing the Binary Classifications of Many Raters
Author(s): Kerrie Nelson* and Don Edwards
Companies: Boston University and University of South Carolina
Keywords: Agreement; Kappa; Binary classifications; Cancer screening
Abstract:

In view of improving breast cancer detection rates, large scale agreement studies have been conducted to evaluate the consistency of cancer ratings of many radiologists. Challenges arise in these studies where a sample of patients undergo two screening tests, resulting in a complex correlation structure between experts' ratings. Here we propose a novel paired kappa measure to compare the agreement between the binary ratings of many radiologists across two cancer screening tests. The proposed kappa appropriately accounts for dependencies between a patient's ratings, corrects for agreement due to chance and is robust to disease prevalence and other flaws often observed in the use of Cohen's kappa. In contrast to existing approaches, the proposed measure can flexibly incorporate large numbers of experts and patients by utilizing the generalized linear mixed models framework. Methods are applied to a recent nationwide breast cancer screening study comparing the ratings of film to digital mammography.


Authors who are presenting talks have a * after their name.

Back to the full JSM 2021 program