|
Abstract:
|
This work sheds light on a surprisingly challenging issue in the analysis and design of multisite randomized trials with heterogeneous treatment effects. The standard precision-weighted estimators suffer from nontrivial but commonly ignored bias when the sites' effectiveness and design features (e.g. size) are correlated, a situation likely in many trials. At the same time, we show that the natural unbiased estimator may be embarrassingly inefficient (to the point of being improved by throwing out data), so a difficult bias-variance tradeoff can arise. We provide theoretical and simulation results to diagnose and manage this tradeoff. These methods are motivated by two iconic multisite trials in education and social welfare, the Head Start Impact Study and Welfare to Work. We discuss in detail how the bias-variance tradeoff plays out in these studies and illustrate how to conduct sensitivity analyses to inform the choice of estimator. The implications are that researchers may need to reconsider how they routinely design and analyze multisite trials.
|
