Detecting Duplicate Homicide Records Using a Bayesian Partitioning Model

• Activity Number: 598
• Type: Topic Contributed
• Date/Time: Thursday, August 7, 2014 : 8:30 AM to 10:20 AM
• Sponsor: Social Statistics Section
• Author(s): Mauricio Sadinle*+
• Companies: Carnegie Mellon
• Keywords: deduplication ; distribution on partitions ; entity resolution ; homicide data ; human rights ; record linkage

Abstract

Finding duplicates in homicide registries is an important step in keeping an accurate account of lethal violence. The task of finding duplicate records in a datafile can be postulated as partitioning the file into groups of coreferent records, where two records are called coreferent if both refer to the same entity. Traditional approaches to duplicate detection output independent decisions on the coreference status of each pair of records, which often leads to non-transitive decisions that have to be solved in some ad-hoc fashion. We present an approach that targets the partition of the file as the parameter of interest, thereby ensuring transitive decisions. Our Bayesian implementation allows to incorporate prior information into the duplicate detection process, which is specially useful when no training data are available, and also provides a proper account of the uncertainty of the duplicate detection decisions. We present an application of this methodology to the detection of killings that were reported multiple times to the United Nations Truth Commission for El Salvador.