Examining the Replication Crisis: The Effect of Underpowered Studies and Publication Bias

• Activity Number: 170 - SPEED: Biopharmaceutical Methods and Application I, Part 1
• Type: Contributed
• Date/Time: Monday, July 29, 2019 : 10:30 AM to 12:20 PM
• Sponsor: Biopharmaceutical Section
• Author(s): Christine M. Orndahl* and Robert A. Perera
• Companies: Virginia Commonwealth University Dept of Biostatistics and VCU Department of Biostatistics
• Keywords: replication; replication crisis; underpowered studies; publication bias; Reproducibility Project

Abstract

The article “Estimating the Reproducibility of Psychological Science” reported alarmingly low reproducibility rates, causing the replicability of science to be questioned. This study examines the role of underpowered studies and publication bias in replication, two explanations that lack attention in discussions of the replication crisis. A simulation study was conducted, generating sets of original studies and their replications. Original studies were underpowered and not replicated until their p-values fell below 0.05 to account for publication bias. The true effect size and difference between the true effect size and the effect size used to determine the sample size for the original study were specified as random variables with a variety of distributions examined. Sample size for the replications were determined using effect size estimates from the corresponding original study. Results from the simulation were compared to results from the article. Simulation results mimicked results from the article, providing evidence that underpowered studies and publication bias are likely causes of the articles findings. Based on these findings, recommendations for researchers are provided.