The Use of Rejection Odds and Rejection Ratios in Testing Hypotheses

• Activity Number: 12 - Bridging BFF (Bayesian/Fiducial/Frequentist) Inference in the Era of Data Science
• Type: Invited
• Date/Time: Sunday, July 30, 2017 : 2:00 PM to 3:50 PM
• Sponsor: IMS
• Author(s): James Berger* and Daniel Benjamin and Thomas Sellke
• Companies: Duke University and University of Southern California and Purdue University
• Keywords: testing ; p-values ; odds ; power ; Bayes factor

Abstract

Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over 50 years. We discuss, as an alternative, the use of the odds of a correct rejection of the null hypothesis to incorrect rejection. Both pre-experimental versions (involving the power and Type I error) and post-experimental versions (depending on the actual data) are considered. Implementations are discussed that range from depending only on the p-value to consideration of full Bayesian analysis. A surprise is that all implementations -- even the full Bayesian analysis -- have complete frequentist justification, thus providing an important instant in which BFF is attainable in testing. Versions of these techniques can be implemented that require only minor modifications to existing practices, yet overcome some of their most severe shortcomings.