456 – P-Values and "Statistical Significance": Deconstructing the Arguments
Multiple Testing and Multiple Modeling Problems with Observational Studies
Stanley Young
CGStat
There is a replication problem in science. Some of the causes are presented. We focus on multiple testing and multiple modeling, MTMM, in this paper. It is impossible to have severe testing if there is no control over MTMM. We focus on observational studies although there is good evidence of problems with experimental studies. A meta-analysis uses statistics coming from base papers to examine/make a claim. We start with a claim coming from a meta-analysis. We use two strategies to evaluate the claim. First, we examine the base papers and count the number of questions at issue. Second, we plot the ranked p-values from the base papers against the integers, 1, 2, …, N, a p-value plot. If the p-values are predominantly small, say <0.05, the claim is supported. If they form a 45-degree line, then the claim is not supported. We usually see a surprising result: a hockey stick figure. The small p-values on the blade of the hockey stick imply a real effect. The large p-values on the handle of the hockey stick imply no effect. W. Edwards Deming would say that asking workers to fix a system where they are successful is doomed. He would identify the replication problem as a management problem, not a worker problem.