Qualitative Framework for Iterative Development of NCSES’s Microbusiness Survey (303209)

Keywords: Establishment survey development, Questionnaire pretesting, Cognitive interviewing

The National Center for Science and Engineering Statistics (NCSES) is developing a new survey, the Microbusiness Innovation Science and Technology (MIST) survey, in response to the National Academy of Science’s Committee on National Statistics (CNSTAT) recommendation to explore ways to measure innovation and research and development (R&D) activities in microbusinesses (companies with fewer than 5 employees). Due to the “micro” nature of the target business population, NCSES expected the survey to have elements in common with a household survey. Therefore, NCSES incorporated into the design and development process a combination of qualitative methods used for household surveys as well as establishment surveys.

The overall survey development approach included exploratory interviews, cognitive interviews, usability tests, and debriefing interviews with microbusiness owners. Westat assisted NCSES with the qualitative research activities. Development began with iterative rounds of exploratory interviews which focused on relevance of topics/questions that have been asked of larger businesses, and if relevant, how best to tailor them for microbusinesses. Feasibility, perceived burden, and motivation to respond were also assessed. Iterative rounds of cognitive interviews focused mainly on question comprehension and response process (steps respondents would need to take to provide the data), while continuing to assess relevance, feasibility, and motivation. Usability testing assessed issues in providing response via the web-based version of the survey. Debriefing interviews focused on response issues that surfaced in the pretest and the pilot test.

This presentation will highlight for survey designers the importance of employing a variety of qualitative methods when developing a new survey for an extremely diverse survey population, and also the need to iteratively revise the content based on respondent motivation and feasibility of providing the requested data.