502 – Innovative Statistical Methods for Complex Survey Data
Inference from Complex Survey-embedded Field Experiments
Robert Ashmead
U.S. Census Bureau
Eric Slud
University of Maryland
Experiments embedded in surveys are a valuable way to test potential survey improvements; however, inference from these experiments can be difficult because one must account for the survey design aspects. Operational issues can restrict possible experimental design choices. In this paper, we discuss a design-based and a randomization-based approach to inference for experiments in an embedded survey. Additionally, some surveys are ongoing and present the option of utilizing data from previous time periods as well as the data from the test period. We discuss and compare an estimator that utilizes data only from the test period with one that compares outcome differences between the study period and a prior period across study treatment groups. Using an example from the American Community Survey (ACS) as well as a simulation, we compare the two inferential approaches and estimators. We find that the permutation-based inferential approach is a flexible and robust method for analyzing embedded experiments and that when the correlation is large between time periods, the difference in time-period differences may provide inferences with greater precision.