189 – Recent Developments in Multiple Testing
Ultra-Fast Permutation-Based Multiple Comparison Adjustment for Weighted Experiments in SAS
David R. Judkins
Abt Associates, Inc.
Permutation-based adjustments for multiple outcomes, treatments or outcomes work well for experiments with complex designs involving varying probabilities of selection and randomization, clustering and stratification. However, SAS Proc MultTest does not accept weights or clusters. By embedding SAS Proc SurveySelect and SurveyReg into SAS macros, it is possible to use the permutation approach on a much wider class of problems. However, this programming approach can be slow, particularly if the experimental sample size is large and the number of permutations is also large. Based on the ideas of Opdyke (2011) for fast permutation tests, I have developed a new SAS macro that is orders of magnitude faster. In this paper, I will discuss the key tricks and share some simulation study results. I will also discuss stabilization of variance estimates when there are few degrees of freedom and how to incorporate this into the fast adjustment.