An Innovative Approach to Cognitive Testing: Using Cognitive Probes in Production CATI Interviews
Samantha Collins
ICF
Georgette Lavetsky
MD Department of Health
Matt Jans
ICF
Traditional cognitive testing typically involves structured qualitative interviews with a small number of participants, and is generally conducted before production interviewing by specially-trained interviewers. This paper asks, "Can we train standardized survey interviewers to administer cognitive probes and obtain information helpful for question revisions?" Our pretest was carried out in the Maryland Behavior Risk Factor Surveillance System (MD-BRFSS) state survey. This innovative pretest method incorporated cognitive probes following each new test questions. The cognitive probes were written as standardized survey questions and programmed directly into CATI, adapting the method to the training and skills of telephone interviewers. Despite major differences in process from traditional cognitive interviewing, we were able to obtain data similar to that obtained by traditional methods, and just as useful for revising questions (i.e., feedback about how respondents interpreted the test questions, and any difficulty they had answering them). We discuss our proof-of-concept and results in the context of pretesting trade-off and evolving approaches to cognitive testing.