An Evaluation of Model-Based and Design-Based Variance Estimators in Completely Randomized Experiments

• Activity Number: 472 - Statistical Methods for Causal Inference
• Type: Contributed
• Date/Time: Wednesday, July 31, 2019 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistics in Epidemiology
• Author(s): Stanley Lubanski* and Peter Steiner
• Companies: University of Wisconsin-Madison and University of Wisconsin
• Keywords: variance estimation; randomized experiment; regression adjustment; average treament effect; heteroscedasticity; experimental design

Abstract

Recently, design-based variance estimators have been proposed to accompany estimates of average treatment effects in completely randomized experiments. While justified by theory, how much of a difference might these design-based variance estimators make in practice when compared to more conventional, model-based variance estimators? This paper seeks to answer this question using simulations to compare the variance estimators (two design-based estimators from Schochet (2015), the conventional regression estimator, and the heteroscedasticity-consistent estimator, HC3) while varying several variance-related factors, including: the scope of inference, outcome heteroscedasticity, balanced/unbalanced design, the presence or absence of covariate adjustment, and sample size. The findings: 1) the design-based estimators are best when heteroscedasticity is caused by idiosyncratic noise and the design is balanced; else they perform similarly to the HC3 estimator and 2) design-based estimators may benefit from a variance-inflation-factor correction.