238 – Instrumentation and Data Quality
Integrating Early Stage Scoping Techniques into Traditional Pretesting Methods: Inside the Development of a Survey on Small Business Lending
Kenneth Herrell
U.S. Census Bureau
Michael Purcell
U.S. Census Bureau
Mary Susan Bucci
U.S. Census Bureau
Snijkers and Willimack (2011) have identified a gap in questionnaire development, where questions may be written without a full understanding of the underlying concept. Subject experts may have a different understanding of key terms and feasibility of data collection than respondents. Cognitive testing of questions written without understanding of the concept can reveal a need for substantial changes. Stettler and Featherston (2010) discuss early stage scoping (ESS): interviews that learn how respondents understand the survey's key concepts, before the questions are written. In practice, however, subject experts often prepare initial questions before the development process begins. While the questions may seem reasonable, testing can reveal that respondents do not understand the conceptual objectives. When this issue is discovered, it may be too late for a full round of ESS. This paper looks at the development of a survey that experienced such issues, and how methodologists and analysts integrated aspects of ESS into subsequent rounds. We also discuss how this may be a way forward in making the practice of early stage scoping a more indelible part of the development process.