HODOR: A Two-Stage Hold-Out Design for Online Controlled Experimentation on Networks

• Activity Number: 27 - SPEED: Statistical Learning and Data Challenge Part 1
• Type: Contributed
• Date/Time: Sunday, August 7, 2022 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Science
• Author(s): Nicholas Alfredo Larsen* and Jonathan Stallrich and Srijan Sengupta
• Companies: North Carolina State University and North Carolina State University and NCSU
• Keywords: network A/B testing; design of experiments; online controlled experiments; lurking variables; spillover effects; causal inference

Abstract

A/B tests are standard tools for estimating the average treatment effect (ATE) in online controlled experiments (OCEs). The majority of OCE theory relies on the Stable Unit Treatment Value Assumption (SUTVA), which presumes the response of individual users depends only on the assigned treatment, not the treatments of others. Violations of SUTVA occur when users are subjected to network interference, e.g. social media platforms. Standard methods for estimating the average treatment effect typically ignore network effects and produce heavily biased results. Additionally, unobserved user covariates, such as variables hidden due to privacy restrictions, that influence both user response and network structure also bias current estimators of the average treatment effect. We demonstrate that network-influential lurking variables bias popular network clustering-based methods, thereby making them unreliable. To address this problem, we propose a two-stage design and estimation technique: HODOR (Hold-Out Design for Online Randomized experiments). We show that HODOR is unbiased for the ATE, has minimizable variance, and can be used even when the underlying network is unknown or uncertain.