Chasing Balance Versus Model Fit When Estimating Propensity Scores

• Activity Number: 648
• Type: Topic Contributed
• Date/Time: Thursday, August 4, 2011 : 10:30 AM to 12:20 PM
• Sponsor: Section on Health Policy Statistics
• Author(s): Beth Ann Griffin*+ and Daniel F. McCaffrey and Daniel Almirall and Rajeev Ramchand
• Companies: RAND Corporation and RAND Corporation and University of Michigan and RAND Corporation
• Keywords: propensity score ; balance

Abstract

Machine-learning methods such as the generalized boosted model (GBM), which utilize multiple trees and adaptive fitting algorithms, have shown considerable promise in yielding propensity score estimates that both achieve balance between treatment and control groups and reduce bias in causal treatment effect estimates. However, guidance on how to implement machine learning techniques to achieve the most accurate treatment effect estimates is still limited. In particular, it is unclear whether GBM-based propensity score models should be tuned to yield the best propensity score model fit or whether they should be tuned to achieve the optimal balance between treatment and control groups. The primary aim of this paper is to compare these two tuning approaches using extensive simulation experiments. We compare the two approaches when estimating both average treatment effects on the population (ATE) and treatment effects on the treated (ATT). We show that, when estimating both ATE and ATT, tuning the GBM to achieve optimal balance is more successful in terms of reducing mean squared error, than tuning the GBM to obtain best model fit.