Study Design and Analysis by Means of Free-Response ROC (FROC) Data: An Evaluation of Clinical Radiological Data

• Activity Number: 359
• Type: Contributed
• Date/Time: Tuesday, August 2, 2011 : 10:30 AM to 12:20 PM
• Sponsor: Biometrics Section
• Author(s): Takayuki Abe*+ and Yuji Sato and Yoshitake Yamada and Kenji Ogawa and Sachio Kuribayashi and Manabu Iwasaki
• Companies: Keio University and Keio University and Keio University and Nippon Koukan Hospital and Keio University and Seikei University
• Keywords: ROC ; free-response ROC (FROC) ; jackknife alternative FROC (JAFROC)

Abstract

The receiver operating characteristic (ROC) method is commonly used in observer performance studies for medical diagnostic devices where every subject is given a score as a diagnostic result. On the other hand, localizing and marking multiple abnormalities on an image is sometimes clinically warranted and, as a result, some images can be given more than one score. Such data is called free-response data, and corresponding FROC curve is defined. One of the frequently applied methods for analyzing such data is the jackknife alternative FROC (JAFROC) method where an AUC-like summary index is used. In this study, an evaluation was made on the effect of the selection of mixed-effects models in the JAFROC when comparing two correlated FROC curves adjusted for multiple observers. It was shown that parsimonious models tend to have higher power in the comparison. Efficiency of some study designs (including balanced incomplete block (BIB) design) was also evaluated. A real example is presented, and used in the evaluation, from a clinical research comparing two diagnostic imaging devices for the detection of pulmonary nodules.