|
Abstract:
|
Kappa is the most commonly used reliability index to study agreement between two raters. Bangdiwala proposed the B-statistic that also corrects for chance agreement as kappa. Though kappa has been widely used, its interpretability and behavior has been questioned and debated. We revisit the paradoxes where kappa has been shown to ill-behave and compare how the B-statistic behaves under similar settings. Paradoxes: (1) Under highly symmetrically unbalanced marginals, a low kappa value observed for high observed diagonal proportion, while B-statistic was more consistently closer to Po. (2) Kappa values are higher for asymmetrical compared to symmetrical marginal imbalances, but this is untrue for B-statistic, thus eliminating the paradox. Under various bias & prevalence settings examined, B-statistics is more stable than kappa, thus we recommend using B-statistic as an alternative to kappa.
|
