|
Abstract:
|
The environment in which an experiment is conducted is unique to each experiment. While the statistical inferences that are drawn from the analysis of experimental data apply only to the environment in which the experiment is conducted, it is almost always the intent of the researcher to apply the results more broadly. The questions then become, will statistically significant results obtained in one environment carry over to others, and if so, how much of a change in environment can be tolerated before inferences are no longer valid? We answer these questions quantitatively by proposing three measures for replicability of statistical inferences: the probability of replicability, the adjusted p-value, and the adjusted confidence interval. Through these measures, we are able to show that larger effect sizes and smaller environmental variability allow for experimental results that are more likely to be replicated. Moreover, if environmental effects are not controlled, replicating a finding may be equivalent to flipping a fair coin, regardless of how well the initial experiment was conducted.
|
