New Insights on the Cognitive Processing of Agree/Disagree and Item-Specific Questions (303336)

Keywords: asking manner, cognitive effort, eye-tracking experiment, question format, web survey

In quantitative social research the use of agree/disagree (A/D) questions, often called Likert items, is a common and a very popular methodological technique to measure different attitudes and opinions of respondents. For instance, in the ANES, the Eurobarometer, and the ISSP this question format is frequently used. Theoretical considerations, however, suggest that A/D questions require an effortful and intricate cognitive information processing. For this reason, Fowler (1995), among others, recommends the use of item-specific (IS) questions since they seem to be less burdensome. In the current study, we investigate cognitive processing of A/D and IS questions in web surveys using eye-tracking methodology. Recording respondents’ eye movements, we are able to investigate how respondents process the survey questions and response categories to determine their information processing. We conducted an eye-tracking experiment with two groups: the first group (n = 44) received three A/D single questions dealing with different political issues (agree/disagree condition). The second group (n = 40) received three IS counterparts dealing with the same question content (item-specific condition). The results of our investigations indicate that IS questions encourage a deeper cognitive processing than A/D questions. Interestingly, the eye-tracking data reveal that this fact is only observable for the processing of the response categories. There are no differences with respect to the question stems; indicating that "question comprehension" seems to be equal. We therefore argue that the observed differences between these two question formats are directly attributed to a more active and more intensive cognitive processing of the IS response categories.