Evaluation of a Confidence Interval Approach for Absolute Inter-Rater Reliability in a Crossed Three-Way Random Effects Model

• Activity Number: 171
• Type: Contributed
• Date/Time: Monday, August 1, 2011 : 10:30 AM to 12:20 PM
• Sponsor: Section on Health Policy Statistics
• Author(s): Joseph C. Cappelleri*+ and Naitee Ting
• Companies: Pfizer Inc. and Boehringer Ingelheim
• Keywords: inter-rater reliability ; intraclass correlation coefficient ; confidence interval; ; analysis of variance ; Monte Carlo simulation; ; generalizability theory.

Abstract

We specify a three-factor random-effects model from an inter-rater reliability study, where the effects of subjects, raters, and items are random. The reliability measure is an intraclass correlation coefficient that measures the absolute agreement of a single measurement from one rater on an item to another rater on the same item. Our objective is to evaluate and illustrate an approximate confidence interval around this intraclass correlation coefficient based on Satterthwaite's approximation (Wong and McGraw. Educational and Psychological Measurement 1999; 59:270-288). In doing so, we performed Monte Carlo stimulations and provided two examples. Overall, coverage of 95% one-sided lower bounds and upper bounds, along with 90% confidence intervals, maintained their true coverage or were slightly conservative. This investigation is the first to validate the methodology.