Confidence Interval Estimation for Inter-Rater Reliability in a Two-Factor Random-Effects Design

• Activity Number: 473
• Type: Contributed
• Date/Time: Wednesday, August 4, 2010 : 8:30 AM to 10:20 AM
• Sponsor: Health Policy Statistics Section
• Author(s): Joseph C. Cappelleri*+ and Kelly H. Zou and Carmen Arteaga and Naitee Ting
• Companies: Pfizer Inc. and Pfizer Inc. and Pfizer Inc. and Boehringer Ingelheim Pharmaceuticals, Inc.
• Keywords: inter-rater reliability ; intraclass correlation coefficient ; confidence interval ; modified large-sample approach ; moments ; Monte-Carlo simulation

Abstract

We specify a two-factor random-effects model from an inter-rater reliability study, where effects of subjects and raters are random. The reliability measure is an intraclass correlation coefficient and our objective is to obtain an approximate confidence interval (CI) around it. We evaluated and illustrated four methods: (1) two-moment approximation, (2) higher-moment based using Pearson's systems of equations, (3) large-sample approximation using a linear combination of the mean squares, and (4) modified large-sample. Their empirical coverage probabilities of the one-sided 95% CIs and two-sided 90% CIs were compared by simulations. The two-moment approach can yield substantially lower-than-desired coverage probabilities on its lower confidence bound. The other three approaches performed better, with the modified large-sample method being the most satisfactory overall.