169 – Making Better Models for Health Studies
Diagnosing the Diagnostic: A Simulation Study of the Omnibus Test for Covariate Balance
Lauren Vollmer
Mathematica Policy Research
When selecting a comparison group for an observational study, accurate measures of balance between the treatment group and the matched comparisons are essential. The omnibus test (Hansen and Bowers 2008) complements covariate-specific assessments with a test of simultaneous covariate balance. However, in practice the omnibus test and covariate-specific diagnostics often disagree. Attempting to reconcile these differences, we conduct a simulation study investigating the omnibus test's power and type I error rates under different data-generating scenarios. Across scenarios, the test's power and type I error rates align with theoretical expectations, with one consistent exception: data sets containing predominantly binary variables. We synthesize these findings, including a comparison of the omnibus test and covariate-specific assessments across scenarios, to offer practical guidance for balance diagnostics.