Agreement by concordance and reader studies
View Presentation View Presentation
*Brandon D Gallas, FDA/CDRH/OSEL/DIAM 

Keywords: reader agreement, concordance, imaging

This talk discusses how concordance generalizes the area under the ROC curve (AUC) when truth is multi-level or unknown. Some examples of multi-level truth are cell expression, time to an event, and truth by an expert panel. When truth is unknown, it is often useful to investigate reader agreement and variability. This talk compares concordance to other agreement measures using simulation, characterizing behavior with respect to data transformations such as rescaling, shifting (bias), and binning. Finally, this talk will describe results from a study where pathologists scored HER2/neu immunohistochemical expression with a continuous scale and a 3-level scale. The results showed that two computer extracted features reduced reader variability, readers used the continuous scale effectively, and the continuous scale increased the precision of the agreement measures.