Improving External Validity of A/B Testing Using Jackknife

• Activity Number: 485 - Decision Making in Tech Giants Through A/B Testing, Prediction and Optimization
• Type: Invited
• Date/Time: Wednesday, July 31, 2019 : 10:30 AM to 12:20 PM
• Sponsor: Quality and Productivity Section
• Author(s): Yu Wang and Somit Gupta and Jiannan Lu* and Ali Mahmoudzadeh and Sophia Liu
• Companies: University of California, Berkeley and Microsoft Corporation and Microsoft Corporation and Microsoft Corporation and Microsoft Corporation
• Keywords: A/B testing; external validity; jackknife; block bootstrap; causal inference

Abstract

On-line experimentation (also known as A/B testing) has become an integral part of software development. In order to timely incorporate user feedback and continuously improve products, many software companies have adopted the culture of agile deployment, requiring A/B tests to be conducted on limited sets of users for a short time period. While conceptually efficient and reasonable, this practice can potentially jeopardize external validity, which is critical for accurate data-driven impact evaluation and decision making. To address this concern, we study external biases for various scenarios in A/B testing, aiming to measure and correct said bias via jackknife re-sampling. In particular, we first highlight that the external bias can be mainly attributed to limited lengths of A/B tests, and then prove that our proposed jackknife estimator could efficiently adjust the first order external bias. We demonstrate the advantages of our proposed methodology by simulated and real-life examples.