An AUC-Like Index for Agreement Assessment

• Activity Number: 26
• Type: Topic Contributed
• Date/Time: Sunday, August 4, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistics in Imaging
• Author(s): Zheng Zhang*+ and Youdan Wang and Fenghai Duan
• Companies: Brown University and Brown University and Brown University
• Keywords: agreement ; AUC ; CCC ; ICC ; ROC curve ; nonparametric estimates

Abstract

The commonly used statistical measures for assessing agreement of readings generated by multiple observers or raters, such as intraclass correlation coe±cient (ICC) or concordance correlation coe±cient (CCC), have well-known dependency on the data's normality assump- tion, hereby are heavily in°uenced by data outliers. Here we propose a novel agreement measure(rank-based agreement index, rAI) by estimating agreement from data's overall ranks. Such non-parametric approach provides a global measure of agreement, regardless of data's exact distributional form. We have shown rAI as a function of the overall ranks of each subject's extreme values. Furthermore, we propose an agreement curve, a graphic tool that aids visualizing extent of the agreement, which strongly resembles the receiver op- erating characteristic (ROC) curve. We further show rAI is a function of the area under the agreement curve. Consequently, rAI shares some important features with the area under of the ROC curve (AUC). Extensive simulation study are included. We illustrate our method with two cancer imaging study datasets.