ReLOOP: Precise Unbiased Estimation in Randomized Experiments Using Observational Auxilliary Data

• Activity Number: 391 - Leveraging Disparate Sources of Data and Machine Learning to Improve Causal Inference
• Type: Topic Contributed
• Date/Time: Tuesday, July 30, 2019 : 2:00 PM to 3:50 PM
• Sponsor: ENAR
• Author(s): Adam Sales* and Johann A Gagnon-Bartsch and Anthony Botelho and Neil T Heffernan and Edward Wu and Luke Miratrix
• Companies: University of Texas At Austin and University of Michigan and Worcester Polytechnic Institute and Worcester Polytechnic Institute and University of Michigan and Harvard University
• Keywords: Causal Inference; Machine Learning; A/B Tests; RCTs; design-based

Abstract

“Design-based” treatment effect estimates from RCTs assume no statistical model other than the experimental design itself. However, when the sample size is small or moderate, these estimates can be imprecise. In contrast, analysis of observational data requires untestable assumptions, chiefly no unmeasured confounding. But often, large observational datasets boast a much larger sample than equivalent RCTs. Sometimes, observational and RCT data coexist within the same database, such as covariate and outcome data from an A/B test—an RCT run within online software—alongside a “remnant” of similar data from users who were not randomized. This paper presents ReLOOP, a novel design-based RCT estimator that incorporates remnant data via machine learning, in order to improve statistical precision. It boasts the same accuracy guarantees as traditional design-based RCT estimators, even in small samples. ReLOOP combines two recent causal methods: rebar, which incorporates remnant data into RCT estimators, and LOOP, which uses machine learning for covariate adjustment in RCTs. We demonstrate ReLOOP in an analysis of A/B tests run within the ASSISTments online mathematics tutor