Comparing Face-to-Face Cognitive Interviewing with Unmoderated, Online Cognitive Interviewing with Embedded and Follow-Up Probing (303216)

Keywords: cognitive interviewing, online cognitive interviewing, unmoderated cognitive interviewing, embedded probing

This study compares cognitive interviewing results using two basic approaches: face-to-face versus online, unmoderated testing. The face-to-face testing relied on follow-up, pre-scripted probes that were asked after all the survey questions had been asked, as well as emergent probes asked during each cognitive testing session. Fifty one questions dealing with the availability and use of paid or unpaid leave were tested. Thirty persons participated separately in each approach: face-to-face and online (N = 60). Because the online testing was completely unmoderated, it did not allow any researcher interventions. Rather, it relied on participants reading and following a script, with “talking out loud” occurring throughout a session. In addition, the online approach compared two approaches for presenting the scripted probes. In one group of 15, the pre-scripted probes were embedded and asked immediately after a target survey question (a question that was thought could cause problems). In the second online group, the same set of pre-scripted probes appeared after all survey questions had been asked. The face-to-face and online groups will be compared on measures such as the number of problems uncovered per question and overall, the types of problems uncovered, quality of response to the probes (e.g., nonresponse and length of response), how well online respondents could remember their responses to probed questions, conclusions drawn from each method, and pros and cons of each method.