The Weighted Brennan-Prediger Coefficient for Multiple Raters in Low Variance Data with Missing Values

• Activity Number: 362
• Type: Contributed
• Date/Time: Tuesday, August 2, 2011 : 10:30 AM to 12:20 PM
• Sponsor: Section on Government Statistics
• Author(s): Mario Jose Nunez*+ and Jamie Doyle and Amy Marshall Richards and Mary Beth Hughes
• Companies: IDA Science and Technology Policy Institute and IDA Science and Technology Policy Institute and IDA Science and Technology Policy Institute and IDA Science and Technology Policy Institute
• Keywords: inter-rater ; agreement ; Finn ; Brennan-Prediger ; reliability ; Krippendorff's Alpha

Abstract

As part of an evaluation of the National Institutes of Health Director's (NIH) New Innovator Award (NIA), the IDA Science and Technology Policy Institute assessed the award application review process, including the degree of inter-rater reliability. During the NIH selection process, applications were judged by three external reviewers on four criteria. After testing several widely used inter-rater reliability measures, I found the measure of reviewer agreement to be fairly low, which was counter-intuitive given the apparent strong agreement in the data. This paper describes an investigation of various inter-rater reliability measures and their specific applications to different data structures. A sensitivity analysis was conducted to compare common agreement coefficients. Based on the award evaluation findings, the author argues that use of a modified version of the Brennan-Prediger coefficient is dependable when data are missing at random, have a small range, and are on an interval scale. The code for the R programming language to run the Weighted Brennan-Prediger coefficient and the proof of equality of Gwet's Brennan-Prediger and Finn coefficients are included.