Experimental Evaluation of Computer-Assisted Human Decision Making: Application to Pretrial Risk Assessment Instrument

• Activity Number: 244 - Recent Advances in Causal Inference with Applications for the Public Good
• Type: Invited
• Date/Time: Tuesday, August 4, 2020 : 1:00 PM to 2:50 PM
• Sponsor: Section on Statistics in Epidemiology
• Author(s): Kosuke Imai* and Zhichao Jiang and James Greiner and Ryan Halen and Sooahn Shin
• Companies: Harvard University and University of Massachusetts, Amherst and Harvard Law School and Harvard Law School and Harvard University
• Keywords: causal inference; machine learning; fairness; principal stratification; randomized experiment; program evaluation

Abstract

Despite an increasing reliance on computerized decision making in our day-to-day lives, human beings still make highly consequential decisions. As frequently seen in business, healthcare, and public policy, recommendations produced by statistical models and machine learning algorithms are provided to human decision-makers in order to guide their decisions. The prevalence of such computer-assisted human decision making calls for the development of a methodological framework to evaluate its impact. Using the concept of principal stratification from the causal inference literature, we develop a statistical methodology for experimentally evaluating the causal impacts of machine recommendations on human decisions. We apply the proposed methodology to the randomized evaluation of a pretrial risk assessment instrument (PRAI) in the criminal justice system. Judges use the PRAI when deciding which arrested individuals should be released and, for those ordered released, the corresponding bail amounts and conditions. We analyze how the PRAI influences judges' decisions.