Implementing Multiple Testing Procedures for Simulation-Based Tests with Bounded Risk

• Activity Number: 78
• Type: Contributed
• Date/Time: Sunday, August 3, 2014 : 4:00 PM to 5:50 PM
• Sponsor: Section on Statistical Computing
• Author(s): Georg Hahn*+ and Axel Gandy
• Companies: Imperial College London and Imperial College London
• Keywords: Benjamini Hochberg ; bootstrap p-value ; false discovery rate ; multiple comparisons ; familywise error rate ; multiple hypothesis test

Abstract

Most multiple testing procedures assume ideal p-values,e.g. the ones of Holm or Benjamini-Hochberg. This talk considers multiple testing under the assumption that the ideal p-value of each hypothesis is unknown and thus approximated using Monte-Carlo(MC) methods. We are interested in obtaining the same (non-)rejections as the ones based on ideal p-values. Established methods for this scenario do not guarantee the correctness of their results. This talk presents MMCTest[1], an algorithm giving the same classification as the one based on ideal p-values with pre-specified probability. The idea of MMCTest is generalized into a new framework for MC multiple testing. We establish conditions on both a generic algorithm to draw samples and an arbitrary testing procedure which guarantee that the (non-)rejections obtained through MC tests only are identical to the ones based on ideal p-values. The framework can be used to improve established methods without proven properties in such a way as to yield certain theoretical guarantees on their results. [1] Gandy,A.& Hahn,G.(2014) "MMCTest - A Safe Algorithm for Implementing Multiple Monte Carlo Tests" Scandinavian Journal of Statistics.To appear