Online Program Home
My Program

Abstract Details

Activity Number: 568 - Experimentation at Scale: Current Challenges in A/B Testing
Type: Topic Contributed
Date/Time: Wednesday, July 31, 2019 : 2:00 PM to 3:50 PM
Sponsor: Section on Statistics in Marketing
Abstract #301697 Presentation
Title: Experimentation at Scale: Current Challenges in A/B Testing
Author(s): Martin Tingley* and Eytan Bakshy* and David Afshartous* and Kathy Zhong*
Companies: Amazon and Facebook and Google and Netflix
Keywords: Experimentation; A/B testing; Bayesian hypothesis testing; sequential testing; power

Large-scale A/B experimentation conducted by technology companies (both large and small) has an impact on the daily lives and decisions of billions of people around the globe, as nearly every aspect of the internal operations and consumer experience of companies like Facebook, Google, Amazon, and Netflix have been tested in some manner.

Experimentation in this space operates at vast scale (in terms of both sample sizes and number of tests) and can require rapid decision making. Developing and improving the associated statistical methods for experimental design and analysis is an ongoing topic of applied research at each of the companies represented by the panelists. Example topics include optimal stopping rules for sequential tests, fast identification of negative test experiences, robustness to outliers, Bayesian approaches to A/B testing and overall decision making, and development of statistical methods to better detect small yet meaningful effect sizes amidst noisy data.

This panel session will discuss the current state and limitations of experimentation in this space, and aims to foster more open dialogue and engagement between private sector and academic researchers.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2019 program