Can Web Probing Be Used to Pretest Survey Questions? (303542)

Keywords: web probing, pretesting, cognitive interviewing

Asking probing questions via web probing has recently been advocated as a promising method for evaluating survey questions, both as a post-survey assessment of item validity and as a pretesting method to identify problems in questions. In comparison to standard f2f cognitive interviewing, the increasing availability of internet nonprobability panels allows a time and resources saving recruitment of respondents and a realization of larger sample sizes.

In the present study, we examine whether web probing can replace standard cognitive interviewing by asking: Does web probing produce similar results as f2f cognitive interviewing? The study reports on the results of 508 respondents drawn from a nonprobability online panel who completed an online survey including four questions from the ISSP 2013/2014. The same questions had been tested previously via f2f cognitive interviewing. It is examined whether web probing and cognitive interviewing identify similar problems in these four questions. Initial findings indicate that web probing and cognitive interviewing detect similar problems. However, web probing itself has some limitations. Practical implementations and directions for future research are discussed.