Online Program Home
My Program

Abstract Details

Activity Number: 485 - Decision Making in Tech Giants Through A/B Testing, Prediction and Optimization
Type: Invited
Date/Time: Wednesday, July 31, 2019 : 10:30 AM to 12:20 PM
Sponsor: Quality and Productivity Section
Abstract #300364 Presentation
Title: Improving External Validity of A/B Testing Using Jackknife
Author(s): Yu Wang and Somit Gupta and Jiannan Lu* and Ali Mahmoudzadeh and Sophia Liu
Companies: University of California, Berkeley and Microsoft Corporation and Microsoft Corporation and Microsoft Corporation and Microsoft Corporation
Keywords: A/B testing; external validity; jackknife; block bootstrap; causal inference

On-line experimentation (also known as A/B testing) has become an integral part of software development. In order to timely incorporate user feedback and continuously improve products, many software companies have adopted the culture of agile deployment, requiring A/B tests to be conducted on limited sets of users for a short time period. While conceptually efficient and reasonable, this practice can potentially jeopardize external validity, which is critical for accurate data-driven impact evaluation and decision making. To address this concern, we study external biases for various scenarios in A/B testing, aiming to measure and correct said bias via jackknife re-sampling. In particular, we first highlight that the external bias can be mainly attributed to limited lengths of A/B tests, and then prove that our proposed jackknife estimator could efficiently adjust the first order external bias. We demonstrate the advantages of our proposed methodology by simulated and real-life examples.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2019 program