Using Iterative, Small-Scale Quantitative and Qualitative Studies: A Review of 15 Years of Research to Redesign a Major Federal Government Survey (303157)

Keywords: measurement error, redesign, multi-method

Measurement error can be very difficult to assess and reduce. However, the longer a survey is in use, the more established the time series becomes, and any change to a questionnaire risks a break in that time series. Thus, a comprehensive survey redesign is generally approached with caution. This paper documents a long-term research project begun in 1999 in which a key module of a major federal survey – the Current Population Survey health insurance module – was redesigned over the course of 15 years through a series of small, iterative tests, both qualitative and quantitative. The aim was to identify features of the questionnaire that could be contributing to measurement error, explore and modify those features, and test against the status quo to assess empirical evidence for improvements. After a decade of this testing, a fundamentally redesigned questionnaire was crafted, a formal pretest was conducted in 2009 and the basic approach was found to be sound. Minor refinements were made and a large-scale split-ballot experiment was conducted in March 2010 with promising results. While that study was in the field, the Affordable Care Act was passed. The draft redesign was adapted with questions specific to health reform and tested, which proved successful. A large-scale field test in 2013 produced results favorable to the redesign, and it was officially implemented in March 2014. While the particular subject of this research is health insurance, the general approach (a coordinated series of small, iterative tests), along with the specific tests and methods employed, are not unique to health insurance. Furthermore, the particular questionnaire design features of the CPS health module that were found to be most problematic are used in many other major surveys on a range of topic areas. Thus, the paper concludes with a discussion of the broader implications of this research for other topic areas and for survey redesign efforts in general.