Assessing Agreement with Relative Area Under the Coverage Probability Curve

• Activity Number: 26
• Type: Topic Contributed
• Date/Time: Sunday, August 4, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistics in Imaging
• Author(s): Huiman Barnhart*+
• Companies: Duke University
• Keywords: Agreement ; Reliability ; Reproducibility ; Coverage probability ; Measurement error

Abstract

Quantitative biomarkers (QIB) from medical images are becoming important tools for clinical diagnosis, monitoring and treatment in radiology. Because errors are inherent in measurement procedure, one must understand the degree of measurement error before QIB's use in practice. An intuitive way to visualize measurement error is to examine the differences between measurements made on the same subject. Acceptable measurement error should correspond to having small/acceptable differences of measurements for majority of the subjects. This corresponds to requiring a large proportion of pairwise absolute differences to fall within a pre-specified acceptable difference, an approach of coverage probability. We propose to use the coverage probability curve, i.e., the cumulative distribution of the absolute difference between two measurements on the same subject, to provide a full spectrum of measurement error. A relative area under the coverage probability curve (RAUCPC) is proposed to summarize the overall agreement which can be used for comparison of different methods/labs/observers. Simulation studies and QIB examples from radiology are used for illustration of the methodology.