An Empirical Evaluation and Guideline of the R-Value for Replicability Analysis Using the Cochrane Database of Systematic Reviews

• Activity Number: 76 - Contributed Poster Presentations: Section on Statistics in Epidemiology
• Type: Contributed
• Date/Time: Monday, August 3, 2020 : 10:00 AM to 2:00 PM
• Sponsor: Section on Statistics in Epidemiology
• Author(s): Mengli Xiao* and Lifeng Lin and James S. Hodges and Yong Chen and Haitao CHU
• Companies: University of Minnesota and Florida State University and Division of Biostatistics, University of Minnesota and University of Pennsylvania and the University of Minnesota Twin Cities
• Keywords: Cochrane Database of Systematic Reviews; meta-analysis; replicability; r-value; systematic review

Abstract

Replicability is often assumed a priori for the studies included in Systematic reviews and meta-analyses (SRMAs), where evidence from multiple studies is synthesized for clinical guidelines. However, this important assumption is rarely investigated in practice,and lack of replicability can affect the accuracy of resulting clinical guidelines. Given a growing concern about replicable experiments,a P-value-based statistic called the r-value was recently proposed to measure an SRMA’s replicability. Although the r-value was applied to a limited subset from a leading source of healthcare evidence, the Cochrane Database of Systematic Reviews (CDSR), it still lacks both an intuitive clinical interpretation and a comprehensive assessment using the CDSR. To fill this research gap, we propose a forward stepwise algorithm for the r-value to obtain a clinical interpretation: the proportion of nonreplicable studies per SRMA,which allows an easy understanding of the degree of nonreplicability. To guide future replicability analysis, we perform large-scale analyses on 520,629 SRMAs both to assess the r-value’s empirical properties and to provide guidelines for clinical interpretation of r-values.