Bayesian Estimation of Program-Specific Impacts in the HPOG Program

• Activity Number: 164 - Social Statistics Speed Session
• Type: Contributed
• Date/Time: Tuesday, August 10, 2021 : 10:00 AM to 11:50 AM
• Sponsor: Social Statistics Section
• Author(s): Stas Kolenikov* and David Ross Judkins
• Companies: Abt Associates and Abt Associates
• Keywords: heterogeneous treatment effects; small area estimation; hieararchical Bayesian modeling; generalized linear mixed models; impact evaluation; labor force training

Abstract

Health Profession Opportunity Grants (HPOG) programs funded by the HHS ACF provide career transition support for low-income adults. We report the methods used in an evaluation to assess the success of these HPOG programs and to provide local, program-specific estimates of impact. Direct estimation of the local average treatment effect consists of simply comparing the means of outcomes for the local treatment and control groups. To overcome small sample sizes typical for most programs, we developed a complementary set of Bayesian estimates of local impacts based on mixed effect models with program-level random effects and random slopes for the treatment indicator. Before preparing Bayesian estimates of local program effects for the second round of grants (HPOG 2.0), we demonstrated the techniques on the previous round (HPOG 1.0). In addition to allowing methods revisions free of p-hacking, we were able to use the posterior distributions for variance components from HPOG 1.0 as priors for HPOG 2.0. Bayesian estimates of impact exhibited less variability than direct estimates did. Bayesian credible intervals were shorter than the confidence intervals at the same coverage level.