Online Program

Study Design Elements That Enhance The Ability To Predict Local Treatment Effects (307872)

Keywords: generalizability, treatment effect heterogeneity, randomized controlled trials

Multisite trials produce rigorous evidence on the average treatment effect across the sites in the trial sample. Depending on how much effects vary across sites and how much of that variation can be explained by observed factors, the results of multisite trials may not be useful in trying to predict a local effect in a site that did not participate in the trial. We conducted simulations based on data from the Head Start Impact Study to assess how well estimators that differed in how they incorporated observed moderators were able to predict local effects. We varied four parameters: number of sites sampled; number of kids per site sampled; strength of observed effect moderators; and degree of unexplained variation in effects across sites. The extent to which estimators using more moderators performed better than those using fewer increased as the observed effect moderation increased, but all estimators performed poorly when the unobserved effect variation was high. The benefits of having a greater number of sites or kids diminished as sample sizes increased. The ability of multisite trials to inform local policy decisions rests on the quality of their data about effect moderation.