Pre-Testing Survey Question Wordings Using Linguistic Resources: The Case of Low-Frequency Wordings (303169)

Keywords: question wording, pre-testing methods, expert reviews, cognitive interviews, split-ballot experiments, question comprehensibility, unfamiliar terms, wording frequency, text corpora, lexical databases

Writing good survey questions is a complex task where several decisions need to be made. Every item can be worded in numerous ways and it is often not clear which is optimal. Certain text features can cause comprehensibility issues that can affect response quality in several ways: increased response burden, longer response times, more satisficing, more item nonresponse and drop-outs, and decreased validity and reliability. To a certain extent, comprehensibility problems in survey questions can be detected with pre-testing and evaluation methods such as expert reviews and cognitive interviewing. Further possibilities to pre-test questions are supported by computer software: Question Understanding Aid (QUAID) employs different psycholinguistic determinants of question complexity and attempts to detect corresponding problems in questionnaire wording. In this paper we discuss the potential of linguistic resources such as text corpora and lexical databases to be used in the questionnaire development process to prevent comprehensibility issues. First, we overview previous uses of computational linguistics resources in questionnaire design and corresponding experiments that study how certain text features affect response quality. Second, we present our procedure that focuses on the issue of low-frequency wordings, which is one of the text features associated with comprehensibility issues. We retrieve word frequencies as estimates of wording familiarity from text corpora and we use wordnets to search for suggestions of alternative wordings. Third, we apply the procedure on three survey questionnaires. Based on linguistic resources we develop low-frequency and high-frequency versions of the same questions. We evaluate the different versions with expert reviews, cognitive interviews, and split-ballot studies on the population. We compare evaluations with word frequencies to evaluate the consistency of results within and between methods.