Split Sample Methods in Observational Studies with Choice of Multiple Hypotheses
Ruth Heller, Technion - Israel Institute of Technology 
Dylan Small, University of Pennsylvania 
*Kai Zhang, University of Pennsylvania 

Keywords: Multiple Comparisons, Split Samples, Sensitivity Analysis

In observational studies with multiple outcomes, the method of sample splitting --- sacrificing a small portion, say 10% of the data to guide design and analysis --- can, in favorable circumstances, yield reduced sensitivity to unmeasured biases by guiding the needed decisions. Current usage of this split sample method focuses on choosing one hypothesis for the analysis. However, in practice, sometimes it is more of interest to test several hypotheses instead. In this study, we investigate a method for choosing multiple hypotheses from the planning sample, and testing these hypotheses on the rest of the sample. We show that the split sample method provides insights on the data and improves the design sensitivity. Simulations are presented and a real health outcomes data set is used for illustration.