Covariate Balance in Propensity Score Models: Much Ado About Nothing?

• Activity Number: 519
• Type: Contributed
• Date/Time: Wednesday, August 6, 2014 : 10:30 AM to 12:20 PM
• Sponsor: Social Statistics Section
• Author(s): Jessica Montgomery*+ and Eun Sook Kim and Jeffrey D. Kromrey and Rheta E. Lanehart and Patricia Rodriguez de Gil and Derrick Saddler and Yan Wang
• Companies: and University of South Florida and University of South Florida and University of South Florida and University of South Florida and University of South Florida and University of South Florida
• Keywords: propensity score ; covariate balance ; estimation error

Abstract

Conventional wisdom states that sample covariate balance is needed to obtain unbiased treatment effect estimates in propensity score models; yet the literature offers little empirical evidence of a relationship between sample covariate balance and the quality of the estimated treatment effect. The present study used simulation to investigate this relationship. The factors investigated include correlation among covariates, the strength of relationships between covariates and both treatment assignment and outcome, the number and reliability of covariates, the magnitude of the population treatment effect, sample size, and accuracy of model specification. Each sample was analyzed for both the degree of covariate balance and estimation error in the treatment effect estimate. Results indicate increased balance only yields improved estimates in naïve models. No relationship is evident between sample covariate balance and estimation error in models that adjust for covariate differences. Results are interpreted in terms of the discrepancy between sample estimates and population parameters, and the potential for sample balance estimates to provide useful information about the quality of propensity score models.