Online Program

Estimating Sensitivity and Specificity for Technology Assessment in Observer Agreement Studies

*Robert M Nishikawa, University of Chicago 

Keywords: ROC, sensitivity, specificity, precision, observer variability

Observer agreement studies and clinical studies are useful to compare two different technologies. The area under the receiver operating characteristic (ROC) curve is often used as the endpoint. However, in clinical practice, observers will operate at a single point on the ROC curve and this will define the sensitivity and specificity of the observer using the technology. In an ROC study, sensitivity and specificity are often estimated based on the ratings that the observer gave each case, which are needed to generate the ROC curve. Unfortunately, because of intra-reader and inter-reader variability, estimates of sensitivity and specificity based on these ratings can lack accuracy and precision. We will demonstrate using an observer study of computer-aided diagnosis in diagnostic mammography that estimates of sensitivity can be either over or under estimated. Further, the statistical power of the experiment is substantially reduced because of increased variability between readers. By simply asking the observer explicitly their recommendation (e.g., biopsy on no biopsy), sensitivity and specificity can be measured directly and the power of the study can be maximized.