|
Abstract:
|
Design replication studies permit comparisons of effect size estimates from observational studies to estimates based on randomized experiments. Prior to 2008, design replication studies used a three-arm design, wherein the control group from a randomized experiment was replaced by a non-randomly selected comparison group. Shadish, Clark, and Steiner (2008) proposed and implemented a four-arm doubly-randomized preference design in which participants were randomly assigned to be in a randomized experiment or a quasi-experiment. Additional randomization handles confounds that could invalidate comparisons based on three-arm designs; however, the four-arm design does not allow for experimental estimation of conditional average treatment effects (ATEs) such as the average treatment effect on the treated (ATT). In this paper, we report on the design and results of a six-arm design replication study that permits experimental and quasi-experimental estimation of the ATE and ATT. By using online platforms, we recruit more participants than have been included in four-arm studies in the past, enabling more precise tests across designs. Results and practical considerations will be discussed.
|
