Online Program Home
  My Program

All Times EDT

Abstract Details

Activity Number: 165 - SLDS CSpeed 2
Type: Contributed
Date/Time: Tuesday, August 10, 2021 : 10:00 AM to 11:50 AM
Sponsor: Section on Statistical Learning and Data Science
Abstract #318847
Title: Trustworthy and Powerful Online Marketplace Experimentation with Budget-Split Design
Author(s): Min Liu* and Vangelis Dimopoulos and Jialiang Mao and Kang Kang
Companies: LinkedIn and LinkedIn and LinkedIn and LinkedIn
Keywords: A/B testing; experimentation; controlled experiment; causal inference; algorithms; online marketplaces
Abstract:

Online experimentation is the gold standard for measuring product impacts and making business decisions in the tech industry. The validity and utility of experiments, however, hinge on unbiasedness and sufficient power. In two-sided online marketplaces, both requirements are called into question. The Bernoulli randomized experiments are biased because treatment units interfere with control units through market competition and violate the “stable unit treatment value assumption”(SUTVA). The experimental power on at least one side of the market is often insufficient because of disparate sample sizes on the two sides. Despite the importance of marketplaces to the online economy, there lacks an effective and practical solution to the bias and low power problems in marketplace experimentation. In this presentation we address this shortcoming by proposing the budget-split design, which is unbiased in any marketplace where buyers have a finite or infinite budget and is more powerful than alternative designs. We share real-world results showing consistently 15x gain in experimental power and removal of market competition induced bias, which can be over 2x the treatment effects.


Authors who are presenting talks have a * after their name.

Back to the full JSM 2021 program