New Methods for Assessing Observer Agreement

• Activity Number: 294
• Type: Contributed
• Date/Time: Tuesday, July 31, 2012 : 8:30 AM to 10:20 AM
• Sponsor: Biometrics Section
• Author(s): Michael Haber*+ and Yi Pan and Jingjing Gao and Lijun Zhang and Zhaohui Qin
• Companies: Emory University and CDC and Emory University and Emory University and Emory University
• Keywords: Coefficient of individual agreement ; Coefficient of individual equivalence ; ChIP-Seq genomic data

Abstract

Observer agreement is usually assessed by Cohen's kappa for categorical data or by the concordance correlation coefficient for quantitative data. These coefficients have limitations, as they depend on the marginal distributions of the measured variables. We review and compare two new coefficients of agreement that do not share these limitations. Both coefficients require replicated measurements on each study subject. The coefficient of individual agreement compares the disagreement between and within observers. The coefficient of individual equivalence compares the inter-observer disagreement to the disagreement expected by chance, i.e., when readings are assigned to observers at random. Both coefficients can be applied to categorical and quantitative data. An example comparing two methods for identifying protein-interacting genomic regions from ChIP-Seq data is used as an illustration.