|
Abstract:
|
Online experimentation is the gold standard for measuring product impacts and making business decisions in the tech industry. The validity and utility of experiments, however, hinge on unbiasedness and sufficient power. In two-sided online marketplaces, both requirements are called into question. The Bernoulli randomized experiments are biased because treatment units interfere with control units through market competition and violate the “stable unit treatment value assumption”(SUTVA). The experimental power on at least one side of the market is often insufficient because of disparate sample sizes on the two sides. Despite the importance of marketplaces to the online economy, there lacks an effective and practical solution to the bias and low power problems in marketplace experimentation. In this presentation we address this shortcoming by proposing the budget-split design, which is unbiased in any marketplace where buyers have a finite or infinite budget and is more powerful than alternative designs. We share real-world results showing consistently 15x gain in experimental power and removal of market competition induced bias, which can be over 2x the treatment effects.
|
