On the Maintenance and Measurement of Inter-Rater Reliability when Documenting Large-Scale Human Rights Violations

• Activity Number: 148
• Type: Contributed
• Date/Time: Monday, August 12, 2002 : 2:00 PM to 3:50 PM
• Sponsor: Social Statistics Section*
• Author(s): Romesh Silva*+
• Affiliation(s): Columbia University
• Keywords: Inter-Rater Reliability ; Human Rights ; Kappa Coefficient

Abstract

We review the statistical challenges faced when documenting large-scale human rights violations. We present practical methods based on recent field experience in Sri Lanka which, when employed, improve the quality of human rights violations data. These improvements in data quality enhance the ability of researchers to analyze the factors, origins, and causes of human rights violations. Furthermore, through a review of current literature, we present the shortcomings of using conventional inter-rater reliability measures (e.g., Kappa Coefficient) in human rights fieldwork. These shortcomings include: 1.) their focus on measuring levels of agreement between two raters but not multiple raters; 2.) the assumption of statistical independence (i.e., that raters are independent of each other); 3.) the assumption that the number of raters stays constant over time; 4.) assumption that raters will identify the same number of violations and that the only difference in reliability between raters arises from misclassification of violations. In considering these shortcomings, we present the desirable statistical properties of reliability measures for data on large-scale human rights violations.