The Undetectable Difference: An Experimental Look at the 'Problem' of p-Values

• Activity Number: 119
• Type: Topic Contributed
• Date/Time: Monday, August 2, 2010 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Education
• Author(s): William M. Goodman*+
• Companies: University of Ontario Institute of Technology
• Keywords: p-value ; significance ; evidence ; inference ; hypothesis testing ; detectable difference

Abstract

Despite continuing assumptions by many scientists and journals that p-values provide a gold standard for inference, counter warnings are not uncommon. The core problem is not p-values, since "p-value < a" could merely signal that a critical value is exceeded. When estimating a parameter, we provide a range (confidence interval). Why, when testing a hypothesis about a parameter (e.g. µ=x) proceed as if "=" entails exact equality? That is not a standard expected for power calculations, where we are satisfied to reject H0 if the result is merely "detectably" different from (exact) H0. This paper explores, with resampling methods, the impacts on p-values if the null hypothesis is defined as a thick or thin range of values. It also examines, empirically, the extent to which the p-value may or may not be a good predictor of the probability that H0 is true, given the data in the sample.