Using Web Probing to Understand the Cognitive Processes Underlying Respondents’ Behavior When Confronted with Check-All-That-Apply and Forced-Choice Questions (303516)

Keywords: web probing, questionnaire design, forced-choice questions, check-all-that-apply questions

Research has shown that the check-all-that-apply and forced-choice question formats do not produce comparable results, but the underlying reasons for these differences are still unclear. This study contributes to filling this gap by the use of web probing to further explore possible causes for the differing answer distributions. Due to its implementation in web surveys, web probing can more easily realize larger sample sizes compared to standard cognitive interviewing.

505 respondents drawn from a nonprobability online panel were randomly assigned to one of four experimental conditions that varied the question format and the order of the response options. The web survey included three questions on different topics. All questions were followed by a general probe and by several specific probing questions. The focus of the online-administered probing questions was not on respondents problems with questions but on how respondents answer questions when presented in the two different formats and to better understand the reasons why response options are left blank. The insights and limitations of this approach as well as practical implications of the findings are discussed.