|
Abstract:
|
Recently, reproducibility is of rapidly growing concerns due to the failure of replication attempts widely perceived in various research areas, which wiped out many results published in high impact journals. Scientists are faced with the tasks of evaluating reproducibility of the reported results. Traditional statistical methods based on i.i.d. assumptions are inappropriate to fulfill precise measurements of reproducibility for modern applications with non-i.i.d. data generated according to general data generating mechanisms. Motivated by the lack of unified formulations, we introduce a general statistical formulation to measure reproducibility, which in principle can be applied to any data generating mechanism. Focusing on study effects for pooled data analyses, we propose the leave-study-out method to estimate the measures of reproducibility based only on the pooled data originally used, without collecting new data. In addition, the theoretical justification and numerical experiments will be provided to demonstrate our method.
|
