Online Program Home
  My Program

All Times EDT

Abstract Details

Activity Number: 188 - Innovative Applications of Design of Experiments
Type: Invited
Date/Time: Tuesday, August 10, 2021 : 1:30 PM to 3:20 PM
Sponsor: Quality and Productivity Section
Abstract #316804
Title: Optimizing the Online User Experience: Design and Analysis of A/B Tests with Bayesian Comparative Probability Metrics
Author(s): Nathaniel T Stevens*
Companies: University of Waterloo
Keywords: A/B testing; design and analysis of experiments; two-group comparisons; practical significance; Bayesian estimation; Markov chain Monte Carlo

As a means to remain competitive, many tech companies are using online controlled experiments (A/B tests) to improve their products, services, and customer experience. Companies like Google, Netflix, Microsoft and Facebook run tens of thousands of experiments, engaging millions of users, each year. Such experiments are typically used to decide whether a particular product variant outperforms one or more alternatives. Relative to traditional DOE applications, the cost of experimental units is much lower and the ease with which data are collected is much greater. This translates into enormous sample sizes that often produce statistically significant p-values, independent of whether the difference between variants is practically meaningful. In this talk we propose the use of comparative probability metrics as an estimation-based alternative to traditional two-sample hypothesis testing. The proposed Bayesian methodology provides a flexible and intuitive means of quantifying the similarity or superiority of one variant relative to another, while accounting for the magnitude of a practically inconsequential difference. A methodology for sample size determination will also be discussed.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2021 program