|
Abstract:
|
We make three main contributions to the analysis and design of multisite trials (randomized block designs) with heterogeneous treatment effects, which are common in education, social policy, and clinical trials. First, we use potential outcomes and a superpopulation framework to precisely describe different potential populations and estimands of interest, which may diverge considerably when effects vary. Second, we introduce a weighted hierarchical model (WHM) to derive consistent estimators of means and covariance components under weak assumptions for any identifiable population. Third, we show that for some natural populations of interest the WHM estimators may be embarrassingly inefficient (to the point of being improved by throwing out data), so a surprisingly difficult bias-variance tradeoff can arise. We provide theoretical tools to diagnose and manage this tradeoff. The methods are illustrated on two iconic multisite trials in education and social welfare. Implications for study design and analysis appear to be profound.
|
