Online Program Home
My Program

Abstract Details

Activity Number: 65 - New-Generation Experimental Design and Causal Inference in High-Tech Companies
Type: Topic Contributed
Date/Time: Sunday, July 29, 2018 : 4:00 PM to 5:50 PM
Sponsor: Quality and Productivity Section
Abstract #329095 Presentation
Title: Two Tales from A/B Testing: The M Error and Partial Identification in Factorial Designs
Author(s): Jiannan Lu* and Yixuan Qiu and Alex Deng
Companies: Microsoft and Purdue University and Microsoft Corporation
Keywords: Design calculation; Replication crisis; Power analysis; Potential outcome; Partial identification
Abstract:

A/B testing is at the front line of technology innovation. In this talk we tell two stories about large-scale A/B testing at Microsoft. First, we consider the M error by Gelman and Carlin (2014), that is, the fact that making decisions based on statistically significant (e.g., p-val < 0.05) results will naturally induce an exaggeration of the actual underlying truth. This is closely related to the everlasting discussion on how to amend (or replace) the traditional null hypothesis significance testing framework, which is widely considered to be behind the recent replication crisis. Second, we consider how to sharpen randomization-based causal inference for factorial designs with binary outcomes, via the partial identification approach. Several simulated and real-life examples will be provided.


Authors who are presenting talks have a * after their name.

Back to the full JSM 2018 program