Evaluation of a Confidence Interval Approach for Relative Inter-Rater Reliability in a Crossed Three-Way Random Effects Model

• Activity Number: 83
• Type: Contributed
• Date/Time: Sunday, July 29, 2012 : 4:00 PM to 5:50 PM
• Sponsor: Health Policy Statistics Section
• Author(s): Joseph Cappelleri*+ and Naitee Ting
• Companies: Pfizer Inc. and Boehringer Ingelheim Pharmaceuticals, Inc.
• Keywords: inter-rater reliability ; relative agreement ; intraclass correlation coefficient ; confidence interval estimation ; analysis of variance ; Monte Carlo simulation

Abstract

We specify a three-factor random-effects model from an inter-rater reliability study, where the effects of subjects, raters, and items are random. The reliability measure is an intraclass correlation coefficient that measures the relative agreement of a single measurement from one rater on an item to another rater on the same item. Our objective is to evaluate and illustrate an approximate confidence interval around this intraclass correlation coefficient based on Satterthwaite's approximation (Wong and McGraw. Educational and Psychological Measurement 1999; 59:270-288). In doing so, we perform Monte Carlo stimulations and provide an illustration. Overall, coverage of 95% one-sided lower bounds and upper bounds, along with 90% two-sided confidence intervals, was generally maintained. This investigation is, to our knowledge, the first to validate the method.