Online Program

Early Stage Scoping in Establishment Surveys: Bridging the Gap between Survey Concepts and Survey Questions

Keywords: questionnaire design, pretesting, early stage scoping, sliding scale approach, establishment surveys, measurement quality, respondent burden

Traditional testing of survey instruments involves drafting a questionnaire and pretesting it with respondents who try to fill it out. New instruments often misfire, as the researcher learns during initial testing that a draft questionnaire fails to use appropriate terms for the respondent or suffers from a naive view of the survey topic. Establishment surveys are prone to suffer early pretesting failures, given the technical nature of data collected and the misleading or inadequate advice of stakeholders. (Willimack et al., ch. 14 in Questionnaire Evaluation and Testing Methods, 2002) A typical problem is that stakeholders ask for data on vague concepts or theoretical constructs, like “service life of R&D”, and authors of the questionnaire must operationalize these into measurable items for respondents. Generally, this earlier step in questionnaire development – moving from concept to question – has not been well-studied or documented, and it is often overlooked by establishment survey developers. (See the forthcoming book, Designing and Conducting Business Surveys, by Snijkers, et al.) To address this void, this paper describes a research tool that bridges this gap between concept and question, thus increasing data quality for new survey items.

We describe a technique we call early stage scoping that uses a "sliding scale approach” to interview respondents prior to questionnaire design. We demonstrate our use of early stage scoping two national surveys of R&D activity—one of the 50 state governments and one of 40,000 businesses. This technique allowed us to develop our questionnaire items efficiently and cheaply using a small number of visits to respondents, supplemented with phone calls to additional respondents.

To assist survey designers in understanding the sliding scale approach, we outline three principles of early stage interviewing. The first principle is "Start fresh." This principle is implemented with a very small number of exploratory interviews to understand the substantive issues from the respondent's point of view. The second principle is to “Learn the respondent's language.” During a second small group of interviews, a paper prototype is used to check the language in a draft of the instrument. The third principle is to "Know thy user, for it is not you." The final set of interviews covers usability and navigation issues for a nearly-complete instrument. The sliding scale refers to the changes across pretest interviews -- the portion of the interview devoted to substantive issues starts near 100 percent and diminishes to near zero across the timeline of interviews, while the portion of the interview dedicated to administrative and navigation issues starts near zero and rises to nearly 100 percent. The paper shows how early stage scoping interviews can utilize these three principles to minimize the resources needed to maximize measurement quality while minimizing respondent burden in establishment surveys.