All Times ET
Keywords: program evaluation, bayesian mixed models, small area estimation
Local Health Profession Opportunity Grants (HPOG) programs, funded by the Administration for Children and Families of the U.S. Department of Health and Human Services, provide education, training, and support services to help transition low-income adults into healthcare occupations. ACF’s Office of Planning, Research, and Evaluation funded an evaluation to assess the success of these HPOG programs and to provide program-specific estimates of impact. Direct estimation of the local average treatment effect (LATE) consists of simply comparing the means of outcomes for the local treatment and control groups to estimate local impacts. Unfortunately, most programs serve too few students to support estimation of local impacts. To overcome those issues, we developed a complementary set of Bayesian estimates of local impacts based on mixed effect models with random effects defined at the program level, and random slopes for the treatment indicator. Before preparing Bayesian estimates of local program effects for the second round of grants (HPOG 2.0), we demonstrated the techniques on the previous round (HPOG 1.0) evaluation data. In addition to allowing methods revisions without fear of accusations of p-hacking, demonstrating the techniques on HPOG 1.0 first allowed us to use the posterior distributions for components of variance as priors for components of variance for HPOG 2.0. As expected from general methodological considerations, Bayesian estimates of impact exhibited less variability than direct estimates did. Bayesian credible intervals were shorter, often by a factor of about 2 to 4, than the confidence intervals at the same coverage level. At the same time, a frequentist empirical Bayes (EB) analysis of the same mixed models produced confidence intervals that were half as long as Bayesian intervals, still, which highlights the importance of properly accounting for the uncertainty in the variance component component estimation that EB methods cannot fully incorporate