Agreement Index: A Novel Measure for Reproducibility and Agreement

• Activity Number: 126
• Type: Contributed
• Date/Time: Monday, August 2, 2010 : 8:30 AM to 10:20 AM
• Sponsor: Biometrics Section
• Author(s): Zheng Zhang*+
• Companies: Brown University
• Keywords: agreement ; AUC ; CCC ; ICC ; kappa statistics ; reproducibility

Abstract

The commonly used statistical measures for agreement between raters or reproducibility of a method, such as kappa statistics, intraclass correlation coefficient (ICC) or concordance correlation coefficient (CCC), are either depending on the rater's or method's marginal distribution (kappa, CCC) or heavily influenced by outliers (ICC, CCC). Here we propose a novel measure of agreement, agreement index (AI), that is derived from the agreement curve. We show that agreement curve strongly resembles the ROC curve and as a consequence, AI shares some features with AUC. The empirical estimate of AI can be derived from a combined rank of each subject's minimum and maximum values, and as such, it is robust against data non-normality or outliers and not influenced by the marginal distribution of the raters or methods. We illustrate our method with two imaging study datasets.