A Causal Replication Framework for Designing and Assessing Replication Efforts

• Activity Number: 486 - Developing the Methodological Foundations for Replication Sciences
• Type: Invited
• Date/Time: Wednesday, July 31, 2019 : 10:30 AM to 12:20 PM
• Sponsor: Social Statistics Section
• Author(s): Peter Steiner* and Vivian Wong
• Companies: University of Wisconsin and University of Virginia
• Keywords: replication; causal inference; research design; treatment effect heterogeneity; reproducibility

Abstract

Given the central role of replication in the accumulation of scientific knowledge, researchers try to replicate seemingly well-established findings. However, results from these efforts have not been promising—replication rates are disappointingly low. Given this "replication crisis," there remains intense debate about what constitutes a (successful) replication and why replication rates are so low. To address these questions, we propose a novel replication framework that allows for a clear definition of a replication study and a systematic discussion of conditions under which results are likely to replicate. Our Causal Replication Framework views replication as a prospective research design and uses the potential outcomes framework to define the (unknown) causal effect of a well-defined intervention. The Framework has the advantage of identifying research design assumptions for the direct replication of results, and shows how different replication designs can be derived and used to evaluate treatment effect heterogeneity. This approach is in contrast to other conceptualizations of replication, which focus on the post-hoc replication of procedures and methods.