A Quality Threshold for Statistical Controls in Observational Studies (QTSC)

• Activity Number: 457
• Type: Topic Contributed
• Date/Time: Wednesday, August 1, 2012 : 8:30 AM to 10:20 AM
• Sponsor: Social Statistics Section
• Author(s): Kenneth Aaron frank*+ and Spiro Maroulis and Minh Duong and Ben Kelcey
• Companies: Michigan State University and Arizona State University and ETS and Wayne State University
• Keywords: statistical controls ; observational studies ; balance ; regression

Abstract

Recent evidence from intra study comparisons shows that estimates from an observational study can approximate estimates from randomized experiments. Critically, the result depends on adequate statistical controls. But little is known about how to characterize the quality of statistical controls for observational studies for which the randomized analogue has not been conducted. We characterize the quality of statistical controls in terms of Frank's (2000) impact of a confounding variable on a regression coefficient. Critically, Frank's impact captures the dual elements of confounding: the relationship between the confound and the outcome as well as between the confound and treatment assignment (the focus of propensity score methods). We then use the non-linear functional relationship between expected impact and sample size in a theoretical randomized experiment to identify a threshold for asymptotic behavior. We use the threshold to evaluate confounds for balance and as a measure of the quality of statistical controls in an observational study; controls are of high quality if they reduce observed impacts below the threshold. We apply our analysis to an estimated negative effect o