True Trend or Just Pretend? Alternative Loss Functions to Reduce Overfitting in Synthetic Controls

• Activity Number: 167 - SPEED: Missing Data and Causal Inference Methods, Part 1
• Type: Contributed
• Date/Time: Monday, July 29, 2019 : 10:30 AM to 12:20 PM
• Sponsor: Health Policy Statistics Section
• Author(s): Alyssa Bilinski* and Laura A Hatfield
• Companies: and Harvard Medical School
• Keywords: synthetic controls; difference-in-differences; loss functions; parallel trends; observational; causal inference

Abstract

The synthetic control method (SCM), a tool for causal inference in observational time-series data, has been described as “[a]rguably the most important innovation in the evaluation literature in the last fifteen years” (Athey and Imbens 2017). SCM constructs a control group from a weighted pool of donor units by selecting the combination that most closely matches the treatment group trajectory (and sometimes covariates) in the pre-intervention period. We argue that SCM shares many of the limitations previously described in difference-in-difference (DID) analysis. These include failure to detect many violations of parallel trends and overfit control groups. These problems lead to biased treatment effects, and neither are addressed by standard SCM inference. We use simulation to demonstrate that these problems are likely to occur even in well-powered studies with a large donor pool. We propose methods to address them including: 1) adjusting the SCM loss function to consider trend stability, and 2) validating weight selection over different levels of time aggregation and alternative covariates. We apply these to empirical studies and compare to existing methods.