Online Program Home
My Program

Abstract Details

Activity Number: 240 - Distributions and Significance
Type: Contributed
Date/Time: Monday, July 30, 2018 : 2:00 PM to 3:50 PM
Sponsor: Biometrics Section
Abstract #330640
Title: The Problems with the Kappa Statistic as a Metric of Inter-Observer Agreement on Lesion Detection Using a Third-Reader Approach When Locations Are Not Pre-Specified
Author(s): Joanna H Shih* and Matthew D Greer and Baris Turkbey
Companies: National Cancer Institute and National Cancer Institute and National Cancer Institute
Keywords: Inter-observer agreement; Kappa; Index of specific agreement; Prostate cancer

Imaging plays an integral role in cancer lesion detection and characterization in oncology practice. For a diagnostic test to gain wide applicability, it is essential to determine its inter-observer agreement. Diagnostic imaging to detect cancer lesions heavily relies on multi-reader studies to measure the inter-observer agreement, and the kappa statistic has been the predominant index of agreement reported in the radiology literature. The purpose of this work is to point out the problem of using kappa to measure reader agreement on lesion detection under the third-reader determined agreement approach and to present two alternative measures of agreement which are more suitable under this setting. The proposed approaches are illustrated using a data set recently collected from a prostate cancer multi-parametric MRI multi-reader study which used the third-reader approach to determine the inter-reader agreement on lesion detection.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2018 program