Online Program Home
My Program

Abstract Details

Activity Number: 663 - Topics in Large-Scale Online Experimentation
Type: Topic Contributed
Date/Time: Thursday, August 2, 2018 : 10:30 AM to 12:20 PM
Sponsor: Section on Statistical Consulting
Abstract #330721
Title: A Decision-Theoretic Approach to A/B Testing
Author(s): David Goldberg* and James Johndrow
Companies: eBay and Stanford University
Keywords: A/B Testing; Decision Theory; p-value

We address the common situation of a firm that is continuously producing potential modifications to their web site. Each modification goes through an A/B test, but the metric of interest is noisy, making it unclear whether it is a good idea to simply ship whenever the metric has a higher value in treatment than control. What statistical analysis should be used? The standard paradigm is to assume a null hypothesis of no change in the metric, and base the decision on a p-value: ship if p is below cutoff, otherwise don't ship. This requires picking a specific p-value cutoff, such as p = 0.10, but the value is typically very ad-hoc. We develop a principled method of selecting the cutoff using decision theory. Our results suggest that this is a promising approach that can yield a practical method of improving A/B testing decisions. Problems with the standard paradigm have been noted before, leading some to propose control of the false discovery rate (FDR). However, it is still necessary to make an arbitrary choice of the level at which to control FDR, and we know of no previous work addressing that question, either for conventional hypothesis testing or FDR.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2018 program