Saturday, November 12
Questionnaire Design
Sat, Nov 12, 1:45 PM - 3:10 PM
Orchid AB
Development and Testing of Item Wording

Combining Pretest Methods to Investigate Comprehension Problems in Attitude Questions: Evidence from Voting Advice Applications (303122)

*Bregje Holleman, Utrecht University, UiL OTS 
Naomi Kamoen, Tilburg University 

Keywords: cognitive interviewing, statistical analysis, 'no opinion,' political attitude questions

In two studies, we investigated how respondents understand attitude questions and how they respond in case of comprehension difficulties. We investigated this in the context of Voting Advice Applications (VAAs): online tools that provide a voting advice based on a comparison between a respondent's answer and the positions of political parties.

Study 1 consists of cognitive interviews with 60 users filling out 30 VAA statements prior to municipal elections in Utrecht. Results show that VAA users encounter comprehension problems for 1 in 5 questions. About two-thirds of these are related to the semantic meaning of the question, covering difficulties with political jargon (e.g., 'dog tax'), or geographical terms (e.g., a specific street in Utrecht). One-third of the comprehension problems are related to the pragmatic comprehension-about the question, often triggered by vague quantifying term (e.g., 'extra'), which make the users realize they lack knowledge about the current state of affairs ('How high is the budget now?'). In case of comprehension problems, VAA users often assume a certain question meaning, and hardly ever proceed by looking for information on the web. Nevertheless, a large majority of the respondents provides a substantive answer (often the middle option). In Study 2, we investigated whether the question characteristics leading to comprehension difficulties in Study 1, lead to more neutral and no opinion-answers when statistically analyzed across a larger set of questions. We performed a statistical analyses of all answers provided by 357,858 VAA respondents who used one of the 34 different municipal VAAs during the Dutch municipality elections in 2014. Results in Study 2 confirm that political jargon, geographical locations and vague quantifying terms are related to more neutral and/or no opinion-answers. We will discuss the methodological advantages of combining methods, and we will outline the implications of our findings for questionnaire design.