Does Better Balance Translate Into Less Bias? Examining Balance Measures and Their Association with Bias in Treatment Effect Estimates

• Activity Number: 648
• Type: Topic Contributed
• Date/Time: Thursday, August 4, 2011 : 10:30 AM to 12:20 PM
• Sponsor: Section on Health Policy Statistics
• Author(s): Elizabeth A. Stuart*+ and Brian K. Lee
• Companies: The Johns Hopkins University and Drexel University
• Keywords: propensity score ; balance

Abstract

A recommended strategy when using propensity score methods is to examine covariate balance as a way to evaluate performance of the method. Balance refers to whether the distribution of a variable differs between treated and comparison groups; a successful propensity score approach should lead to minimal differences in the covariate distributions between these two groups. However, different balance measures have been suggested, and it is unclear whether certain measures are better indicators of bias in the treatment effect estimate than others, and under what conditions. This study examines the correlations of various balance measures with bias in the estimated treatment effect using simulated data. Propensity scores were applied using inverse probability weights. Measures of covariate balance included standardized differences in means, difference in average prognosis scores, Kolmogorov-Smirnov test p-values, and t-test p-values. In all examined scenarios, the average standardized difference in means and the difference in average prognosis scores correlated well with bias in the treatment effect.