Using Targeted Embedded Probes to Quantify Cognitive Interviewing Findings (303170)

Keywords: cognitive interviewing, web surveys, mixed methodology, embedded probing

Cognitive interviewing is one of the most commonly used and affordable question evaluation methods, particularly in large-scale, official statistical systems such as the National Health Interview Survey (NHIS), the National Health and Nutrition Examination Survey, and the Decennial Census. As its methodology has developed, it has become less about “finding problems” with survey questions and more about understanding what specific constructs a survey item actually captures. However, as with any qualitative method, one of the major limitations of cognitive interviewing is the fact that its findings cannot be extrapolated from its purposive sample to the larger, statistical sample of a survey—it provides very “deep” information, but not “wide” information about the actual population that a survey is interested in. One potential way of bridging this gap is the use of target embedded probes—closed- or open-ended questions that can be administered following a survey item that is under evaluation—in fielded surveys. Typically, embedded probes have been administered in survey environments as open-ended questions with text fields that respondents have the option of clicking through and ignoring (in a web survey). While the use of open-ended probes in web-format surveys has shown some promise in discovering problematic survey items and concepts, more specific and directed questions are necessary if the goal is to determine the spread of in- or out-of-scope interpretative patterns across a sample. In order to accomplish this, the qualitative findings of in-depth cognitive interviews must be used to develop targeted, close-ended embedded probes that can be administered directly following a survey item under evaluation. This paper explores the use of these probes, which when used alongside cognitive interviewing, allows for a powerful, mixed-method approach to questionnaire evaluation that delivers methodologists the benefits of both qualitative and quantitative analyses.