Analysis of Small Ensembles of Social Experiments

• Activity Number: 82
• Type: Contributed
• Date/Time: Sunday, August 3, 2014 : 4:00 PM to 5:50 PM
• Sponsor: Social Statistics Section
• Author(s): David Judkins*+
• Companies: Abt Associates
• Keywords: Random slopes ; external validity ; multicenter trials ; treatment-by-center interaction

Abstract

Many randomized trials of social interventions involve randomization of clients (e.g., students, welfare recipients) at each of a small collection of service facilities (e.g., schools, welfare offices). Only rarely are the facilities randomly selected; most often they volunteer or agree to participate after intensive recruitment efforts. Actors at the local facilities mediate the effects of the intervention -- both by the fidelity of their implementation and by the charisma and energy they bring to their organizations. For this reason, generalization of results to broader implementation phases can be hazardous. Nonetheless, if the intervention is successful in the trial, advocates for the intervention will use the results to urge more widespread implementation. If they succeed, the results may be disappointing if it turns out that the intervention only works in the hands of a few skilled actors. This hazard of can be reduced by placing confidence intervals on estimated effects that reflect the variation in effects across sites (the random slopes). The question then arises of how many sites are required to get valid generalization inferences. This paper reviews the literature, lays out some philosophy, and shares some new simulations.