Online Program

Methodological Purity Meets Practical Reality in Embedded Business Survey Experiment Design: Experience and Learning from Australia (307946)

Keywords: Response rate, survey response, experiment, embedded experiment

The Australian Bureau of Statistics (ABS) has a program to improve business survey response rates. While rates are high, so to are costs when businesses delay and need follow-up. Implementation of improvements to survey processes and respondent-facing systems will increase timely self-initiated response, yet recent successes in experiments in household surveys have attracted attention to experiments as a tool to test ideas and make hard decisions. Yet the environment for business survey experiments differs from household surveys, not least because experiments are embedded within existing samples. I will outline how the constraints on experiments embedded in live surveys and within specific survey designs required the ABS to balance methodological purity in experimental design with practical factors on a number of experiments. The benefit of practical design is that experiments are conducted but with a cost to generalizability of findings. Ideally, experiments would be repeated in other surveys but cost pressures encourage rapid scaling instead. Less rigorous methods of close monitoring of whole-sample application of somewhat-tested changes to other similar surveys seems necessary.