Abstract:
|
Recently the science community has suffered from a “reproducibility crisis”. While there are many contributing factors, e.g., publication bias, poorly conducted experiments, another factor could be the error rate control method used. False discovery rate (FDR) is attractive for large experiments when adjusting for multiplicity. While FDR and related methods are valuable statistical additions to the science community, FDR was never intended for final confirmatory analyses. Rather FDR was designed to allow false signals in order to increase the chances that most of the true signals were identified. Unfortunately, this is not the common practice. Instead of using FDR only for initial screening to identify as many “leads” as possible, FDR is used for confirmatory experiments and the results published as rigorously establishing significance. In such cases, it is not surprising that many of the “confirmed” signals are subsequently discovered to to be false. Simulation studies investigate the reproducibility of experiments conducted using FDR and and compare those results to other methods of multiplicity control. A three-tier significance assignment is suggested to mitigate the issues.
|