Pathologist Agreement from Quantitative Measurements: A Pilot Study

• Activity Number: 25 - Medical Devices and Diagnostics Speed Session
• Type: Contributed
• Date/Time: Sunday, August 8, 2021 : 1:30 PM to 3:20 PM
• Sponsor: Section on Medical Devices and Diagnostics
• Author(s): Brandon D Gallas* and Katherine Elfer
• Companies: FDA/CDRH/OSEL/Division of Imaging, Diagnostics, and Software Reliability and FDA/CDRH/OSEL/Division of Imaging, Diagnostics, and Software Reliability
• Keywords: Pathologist agreement; Cross-correlated data; Data sharing; Algorithm assessment

Abstract

*Background* We studied interobserver variability in pathologists who evaluate breast cancer biopsy specimens. Our goal is to create a validation dataset that is fit for a regulatory purpose. *Methods* Pathologist annotators completed data collection tasks via two modalities: an optical light microscope and two digital platforms (slides were scanned at 40X). Pathologists were trained on the clinical task of sTIL density estimation before annotating pre-specified regions of interest (ROIs) across all platforms.*Results* The pilot study accumulated 6,257 sTIL density estimates from 34 pathologists evaluating 64 cases, with 10 ROIs per case. The variability of sTIL density estimates in an ROI increases with the mean; the square roots of the reader-averaged mean-squared differences were 8.3, 17.7, and 40.4 as the sTIL density reference score increased from 0-10%, 10-40%, and 40-100%, respectively. *Conclusions* This pilot study will inform the development of statistical methods, simulation models, and sizing analyses for pivotal studies. We will explore this validation dataset and analysis tools and make them publicly available.