JSM 2011 Online Program

The views expressed here are those of the individual authors and not necessarily those of the JSM sponsors, their officers, or their staff.

Abstract Details

Activity Number: 362
Type: Contributed
Date/Time: Tuesday, August 2, 2011 : 10:30 AM to 12:20 PM
Sponsor: Section on Government Statistics
Abstract - #302911
Title: The Weighted Brennan-Prediger Coefficient for Multiple Raters in Low Variance Data with Missing Values
Author(s): Mario Jose Nunez*+ and Jamie Doyle and Amy Marshall Richards and Mary Beth Hughes
Companies: IDA Science and Technology Policy Institute and IDA Science and Technology Policy Institute and IDA Science and Technology Policy Institute and IDA Science and Technology Policy Institute
Address: 1899 Pennsylvania Ave., NW, Washington, DC, 20006,
Keywords: inter-rater ; agreement ; Finn ; Brennan-Prediger ; reliability ; Krippendorff's Alpha

As part of an evaluation of the National Institutes of Health Director's (NIH) New Innovator Award (NIA), the IDA Science and Technology Policy Institute assessed the award application review process, including the degree of inter-rater reliability. During the NIH selection process, applications were judged by three external reviewers on four criteria. After testing several widely used inter-rater reliability measures, I found the measure of reviewer agreement to be fairly low, which was counter-intuitive given the apparent strong agreement in the data. This paper describes an investigation of various inter-rater reliability measures and their specific applications to different data structures. A sensitivity analysis was conducted to compare common agreement coefficients. Based on the award evaluation findings, the author argues that use of a modified version of the Brennan-Prediger coefficient is dependable when data are missing at random, have a small range, and are on an interval scale. The code for the R programming language to run the Weighted Brennan-Prediger coefficient and the proof of equality of Gwet's Brennan-Prediger and Finn coefficients are included.

The address information is for the authors that have a + after their name.
Authors who are presenting talks have a * after their name.

Back to the full JSM 2011 program

2011 JSM Online Program Home

For information, contact jsm@amstat.org or phone (888) 231-3473.

If you have questions about the Continuing Education program, please contact the Education Department.