All Times ET
Virtual
Effect of Simultaneous Error Rates on Reproducibility (304202)
Melinda McCann, Oklahoma State University*Scott Richter, University of North Carolina at Greensboro
Keywords: reproducibility, simultaneous inference, FWE, FDR, multiplicity control
Recently the science community has suffered from a “reproducibility crisis”. While there are many contributing factors, e.g., publication bias and/or poorly conducted experiments, another factor could be the method of error rate control. False discovery rate (FDR) is attractive for large experiments when adjusting for multiplicity. However, while FDR and methods for controlling FDR are valuable statistical additions to the science community, FDR was never intended for final confirmatory analyses. Rather FDR was designed to allow for a certain percentage of the discoveries to be erroneous, i.e., false, thereby increasing the chances of identifying more true signals. "Unfortunately, instead of using FDR only for initial screening, FDR is used for confirmatory experiments, and the corresponding results published as establishing significance. In such cases, it is not surprising that many of the “confirmed” signals are subsequently discovered to be false. Simulation studies investigate the reproducibility of experiments conducted using FDR and compare those results to other methods of multiplicity control. A three-tier significance assignment is suggested to mitigate the issues.