Conference Program Home
  My Program

All Times EDT

Abstract Details

Activity Number: 27 - SPEED: Statistical Learning and Data Challenge Part 1
Type: Contributed
Date/Time: Sunday, August 7, 2022 : 2:00 PM to 3:50 PM
Sponsor: Section on Statistical Learning and Data Science
Abstract #322310
Title: HODOR: A Two-Stage Hold-Out Design for Online Controlled Experimentation on Networks
Author(s): Nicholas Alfredo Larsen* and Jonathan Stallrich and Srijan Sengupta
Companies: North Carolina State University and North Carolina State University and NCSU
Keywords: network A/B testing; design of experiments; online controlled experiments; lurking variables; spillover effects; causal inference

A/B tests are standard tools for estimating the average treatment effect (ATE) in online controlled experiments (OCEs). The majority of OCE theory relies on the Stable Unit Treatment Value Assumption (SUTVA), which presumes the response of individual users depends only on the assigned treatment, not the treatments of others. Violations of SUTVA occur when users are subjected to network interference, e.g. social media platforms. Standard methods for estimating the average treatment effect typically ignore network effects and produce heavily biased results. Additionally, unobserved user covariates, such as variables hidden due to privacy restrictions, that influence both user response and network structure also bias current estimators of the average treatment effect. We demonstrate that network-influential lurking variables bias popular network clustering-based methods, thereby making them unreliable. To address this problem, we propose a two-stage design and estimation technique: HODOR (Hold-Out Design for Online Randomized experiments). We show that HODOR is unbiased for the ATE, has minimizable variance, and can be used even when the underlying network is unknown or uncertain.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2022 program