Introduction: Location bias occurs when a reader detects a false lesion (e.g. non-cancerous lesion) in a subject with disease and the falsely detected lesion is considered a true positive. This can occur in two ways: the reader does not detect the true lesion and only sees false lesions (type I), or the reader detects both the true lesion and false lesions but assigns greater suspicion to the false lesions (type II).

Methods: In this study we examined the effect of location bias in two large MRMC ROC studies, a breast cancer and lung cancer screening studies. Our study had five objectives: 1. Examine the frequency of location bias by reader and modality, 2. Observe the distribution of confidence scores for each implemented method, 3. Identify factors associated with the prevalence of location bias, 4. Compare readers’ AUC estimates using the different methods, and 5. Examine the association between the effect size (difference in readers’ AUC between the two modalities) and the difference in the frequency of location bias of the modalities.

Results: Location bias occurred in 15-20% of cases overall, but varied among readers from 3 to 26%. Type I bias was more common in the breast cancer screening study, while type II bias was more common in the lung cancer screening study.

The ROI-ROC and FROC methods are two approaches for correcting for location bias. The methods correct for type I bias by using a confidence score of zero for missed lesions (rather than the confidence score of a false positive); they correct for type II bias by using the confidence score assigned by the reader to the true lesion (rather than to a higher rated false positive). The correction to the confidence score is usually larger for type I bias, shifting confidence scores to zero. However, the overall spread of readers' confidence scores was maintained after the correction.

Readers with higher false positive rates had significantly more location bias, particularly type II bias, with correlations of 0.72 for the breast cancer study and 0.54 for the lung cancer study. Readers’ sensitivity was not associated with the frequency of location bias.

The magnitude of the correction to the ROC area estimate is strongly correlated with the frequency of the bias. The correction reduces the magnitude of estimate of the ROC area compared with the uncorrected estimate.

When comparing two modalities’ ROC areas, the effect size depends on the difference in the frequency of location bias between the two modalities. When the difference in frequency of bias is small, the effect size is similar whether the location bias is corrected for or not. However, when the frequency of location bias is dissimilar, failure to correct for the location bias favors the modality with higher FPR. In some instances, correcting for the bias increased the effect size, while in others the correction decreased the effect size or reversed the direction of the effect.

Discussion: Although issues associated with location bias have been described previously and statistical methods to correct for the bias have been available for some time, investigators of MRMC studies seldom discuss the bias or how they handled the bias in their analysis. We conducted a small review of published MRMC studies. We reviewed 27 articles published in Radiology and 34 in AJR in 2014. In 17 of these studies, we felt the potential for location bias existed. One study used FROC, 7 used some form of ROI-specific ROC, and 9 used neither method. In general, we found that the methodology for addressing the detection of multiple lesions and potential location bias was seldom addressed.

Others (Dendumrongsup 2014) have also addressed the poor quality of data reporting in MRMC studies. They found that the large majority of studies included less than 10 readers. A small number of readers can easily skew study results when location bias is not taken into account, as we presented in our study.

Our study demonstrates the necessity of adjusting for location bias to obtain statistically valid results. From a clinical standpoint, however, the necessity may vary. In the breast cancer example, the current practice dictates that a radiologist correctly locate a potentially cancerous lesion to be biopsied. If the radiologist locates a lesion that is not cancerous, but present in a cancerous breast, it does not help the patient, and in fact can be detrimental. In the lung cancer study, however, if a suspicious lesion is detected on an x-ray, a CT scan of both lungs is performed. In this case, the radiologist’s ability to correctly locate the lesion is less important, and perhaps location-bias corrections are less necessary.

In conclusion, location bias can have an impact on the results of a study, particularly if only a few readers are used or if the modality influences the frequency of location bias. In order to avoid spurious results, location bias adjustment is recommended.