Estimating Cross-Site Impact Variation in the Presence of Heteroscedasticity

• Activity Number: 141
• Type: Contributed
• Date/Time: Monday, August 5, 2013 : 8:30 AM to 10:20 AM
• Sponsor: Social Statistics Section
• Author(s): Kristin Porter*+ and Howard S. Bloom and Michael J. Weiss and Stephen Raudenbush
• Companies: MDRC and MDRC and MDRC and The University of Chicago
• Keywords: impact variation ; heteroscedasticity ; randomized control trials ; program evaluation

Abstract

Important questions arise when using data from a multi-site RCT to estimate the magnitude of cross-site impact variation and assess statistical significance. Some of these questions arise from the likelihood that variation in individual outcomes may be different for the treatment and control groups and/or for different sites. For example, when impacts vary across individuals, the variance of individual outcomes (i.e. the residual variance) in the treatment group differs from that in the control group. In addition, when different populations and local conditions are represented by different sites, the variance of individual counterfactual outcomes differs across sites. This can be exacerbated if the extent of individual variation in impacts also varies across sites. If not properly accounted for, these two forms of heteroscedasticity can lead to bias and incorrect statistical inference when estimating cross-site impact variation, particularly when sample sizes within sites are small. Using statistical theory and simulations, we compare estimators of cross-site impacts in the presence of heteroscedasticity, evaluating the risks of estimating too few or too many residual variances.