Using School Lotteries to Evaluate the Value-Added Model

• Activity Number: 437
• Type: Contributed
• Date/Time: Tuesday, August 6, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Government Statistics Section
• Author(s): Jonah Deutsch*+
• Companies: The University of Chicago
• Keywords: Value-Added Models ; School lotteries ; Causal inference

Abstract

This paper empirically tests the central assumption of Value-Added (VA) models: that school assignment is not confounded with unobserved components of students' test scores. I use a charter school lottery to identify school effects, and then compare this "experimental" estimate to that of a school VAM, which is estimated from all students in a large district. I create and demonstrate an innovative approach which shows that, under a common exclusion restriction, the effect of winning the lottery can be expressed as a function of school effects from the VA model. Estimates of school effects from VA can then be used to generate a VA-based estimate of the lottery effect, which will be unbiased if VA is as well. In reading, the VA-based estimates are extremely close to the experimental ones. In math, while the difference is not statistically significant, the magnitudes and direction are also consistent with a selection bias scenario. I test a number of different VA models, including those most common in the literature, and also test whether aggregating over years and/or grade levels impacts the accuracy of the models. This paper also discusses the implications my test has for teacher V