Do Comparison Group Methods Replicate Experimental Impact Estimates in an Education Setting? A Charter School Evaluation Example

• Activity Number: 162
• Type: Topic Contributed
• Date/Time: Monday, August 2, 2010 : 10:30 AM to 12:20 PM
• Sponsor: Social Statistics Section
• Author(s): Kenneth Fortson and Natalya Verbitsky-Savitz*+ and Philip Gleason and Emma Ernst
• Companies: Mathematica Policy Research, Inc. and Mathematica Policy Research, Inc. and Mathematica Policy Research, Inc. and Mathematica Policy Research, Inc.
• Keywords: causal inference ; quasi-experimental designs ; education ; charter schools ; program evaluation

Abstract

Randomized control trials (RCTs) are widely considered the gold standard in evaluating the impacts of a program. However, an RCT is not always feasible, in which case quasi-experimental designs (QEDs) may be used. A popular class of QEDs uses a non-randomly selected comparison group to represent what would have happened to the treatment group had they not participated in the program. Under certain assumptions, comparison group methods can produce unbiased impact estimates; however, these assumptions are generally untestable in practice. This study assesses the validity of regression-based (ordinary least squares and fixed-effects) models and matching (propensity score and exact matching) approaches by using data from a large-scale RCT of charter schools and administrative school district data to test how closely findings from comparison group designs replicate those from the RCT.