On the Reproducibility of Psychological Science

• Activity Number: 565 - JASA Applications and Case Studies
• Type: Invited
• Date/Time: Wednesday, August 1, 2018 : 2:00 PM to 3:50 PM
• Sponsor: JASA, Applications and Case Studies
• Author(s): Val Johnson* and Richard Payne and Tianying Wang and Alex Asher and Soutrik Mandel
• Companies: Texas A&M University and Texas A&M University and Texas A & M University and Texas A&M University and Texas A&M University

Abstract

Investigators from a large consortium of scientists recently performed a multi-year study in which they replicated 100 psychology experiments. Although statistically significant results were reported in 97% of the original studies, statistical significance was achieved in only 36% of the replicated studies. This article presents a re-analysis of these data based on a formal statistical model that accounts for publication bias by treating outcomes from unpublished studies as missing data, while simultaneously estimating the distribution of effect sizes for those studies that tested non-null effects. The resulting model suggests that more than 90% of tests performed in eligible psychology experiments tested negligible effects, and that publication biases based on p-values caused the observed rates of non-reproducibility. The results of this re-analysis provide a compelling argument for adopting statistical summaries of evidence that account for the high proportion of tested hypotheses that are false.