Exploring Pathologist-Pathologist Agreement as a Baseline for Algorithm-Pathologist Agreement

• Activity Number: 16 - Assessing Agreement and Reader Reliability in Medical Imaging Analysis
• Type: Topic Contributed
• Date/Time: Sunday, August 7, 2022 : 2:00 PM to 3:50 PM
• Sponsor: Biopharmaceutical Section
• Author(s): Brandon D Gallas*
• Companies: US FDA, CDRH/OSEL/Division of Imaging, Diagnostics, and Software Reliability
• Keywords: Agreement; Quantitative Data; Reader Study; biomarker; calibration

Abstract

Expert clinical annotations are a common reference standard for training and testing artificial intelligence and machine learning (AI/ML) models. Depending on the task, these annotations can be fraught with biases and variability. We collected biomarker annotations (density values: 0-100%) from multiple pathologists on specific regions of interest (ROIs) nested within independent cases. After exploring the data, we found the ROIs elicit different levels of reader variability and this variability exhibits a complicated relationship with the mean that cannot be stabilized with a transformation. Hence the data are not independent and identically distributed according to a normal distribution, and the data cannot be pooled and analyzed by standard quantitative analysis methods, like correlation, ANOVA, and regression. This led us to investigate the use of thresholds and ordinal rank-based methods, such as ROC analysis, for assessing pathologist agreement and AI/ML performance. The thresholds allow us to understand agreement at specific biomarker density values (calibration) while the ordinal rank-based methods sidestep distributional assumptions of the data.