Analysis of Multi-Reader Studies of Accuracy in Detecting and Localizing Multiple Disease Targets

• Activity Number: 494
• Type: Topic Contributed
• Date/Time: Wednesday, August 1, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Biopharmaceutical Section
• Author(s): Andriy Bandos*+
• Companies: University of Pittsburgh
• Keywords: Multi-reader imaging studies ; detection and localization task ; free-response ROC analysis ; clustered ROC analysis ; diagnostic accuracy

Abstract

Contemporary diagnostic tasks often focus on detection and localization of multiple targets per subject (e.g., nodules per chest image). Evaluation of diagnostic performance in detection-localization tasks requires extensions of the conventional ROC analysis; the two most commonly used extensions are free-response ROC (FROC) and Region of Interest (ROI) approaches. Since in many areas diagnostic results incorporate interpretations by a trained human observer, diagnostic systems are often evaluated in a multi-reader setting. Multi-reader studies generate additional analytical challenges ranging from defining a practically relevant summary index and up to the reliable statistical analysis. In addition, problems existing in a single-reader assessment of detection-localization accuracy -- such as variable length of the performance curves, complex covariance structure of the data, and possible informativeness of the cluster sizes -- become more pronounced in a multi-reader setting. In this talk we will discuss the approaches for summarizing multi-reader performance in a detection-localization task, illustrate properties of the statistical analysis, and offer practical recommendations.