Mixed Designs for Multi-Reader ROC Imaging Studies

• Activity Number: 494
• Type: Topic Contributed
• Date/Time: Wednesday, August 1, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Biopharmaceutical Section
• Author(s): Nancy A. Obuchowski*+ and Stephen Hillis and Brandon Gallas
• Companies: Cleveland Clinic Foundation and University of Iowa and FDA/CDRH/OSEL/DIAM
• Keywords: Multi-reader imaging studies ; diagnostic test accuracy ; ROC analysis ; MRMC study design

Abstract

Diagnostic accuracy of imaging tests is often measured and compared by estimating the ROC curves of multiple readers interpreting these tests. These multi-reader studies often use a fully-crossed design, where patients are imaged with one or more tests, and readers interpret all patients' images. While this design efficiently reduces the number of patients and readers required, it demands a large number of interpretations per reader. Mixed designs, where different readers interpret the test results of different patients, have been proposed to reduce the number of reader interpretations by retaining the patient and reader pairings across tests while reducing the correlations. We propose several test statistics for comparing readers' ROC curves in the mixed multi-reader design. A simulation study was performed using the Roe-Metz model to evaluate and compare the type I error rate, power, and CI coverage of the proposed statistics. This work was motivated by an imaging trial assessing the accuracy of readers with and without a computer aided detection (CAD) device for detecting breast cancer.