A Model-Based Approach for Modeling Agreement in Ordered Classification Scales

• Activity Number: 46
• Type: Contributed
• Date/Time: Sunday, August 4, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Biometrics Section
• Author(s): Kerrie Nelson*+ and Don Edwards
• Companies: Boston University and University of South Carolina
• Keywords: agreement ; ordinal data ; diagnostic test ; kappa statistic ; classification

Abstract

Many diagnostic tests involve the classification of a patient by a medical expert using an ordered categorical scale. Such tests involve subjectivity on the part of the expert due to the necessity to interpret imperfect diagnostic test results, leading to discrepancies between experts' classifications, even in common diagnostic procedures such as mammography. This has motivated many large-scale studies to examine levels of agreement between experts in common diagnostic settings and to investigate if factors such as rater experience affect consistency of ratings. However, limited statistical methods exist to assess agreement in large-scale studies. In this talk, we briefly introduce a model-based, chance-corrected approach for assessing agreement in large-scale agreement studies that also allows for examining the impact of subject- and rater-related characteristics on agreement.