Optimizing Survey Items by Studying Effects of Interviewer Deviations in Question Reading (303120)

Keywords: behavior coding, interviewer-respondent interaction coding, interviewer deviations, pre-test methods, standardized interviewing

In this paper, we show that by looking specifically at types of rewording, many more suggestions for improvement of question wording can be revealed than in mainstream behavior coding studies focusing only on differences between major and minor deviations. We start with a literature review on interviewers' deviations and its consequences for response quality. Then, two studies are presented.

In the first study, we analyze CATI interviewer-respondent interactions of the 2006 Nebraska Annual Social Indicators Survey. This study shows that interviewers add, substitute, and omit words of items. More specifically, it is found that interviewers sometimes spontaneously add numbers to response options when administering the question. Due to this change in question reading, respondents appeared to be better able to formulate an answer. In the second study, effects of explicitly numbered response options on response behavior are tested. An experiment is conducted using the questionnaire of the European Social Survey round 5. It is found that more mismatch answers are given when numbers are not read by the interviewer. However, no significant differences are found between the experimental conditions. For both conditions it is found that respondents answer the questions more often in terms of words than using a number. A possible explanation for the differences between the two studies is that in study 1 the numbers were spontaneously added to questions with (relatively difficult) nominal response options, whereas in study 2 the numbers were added to questions with (relatively easy) ordinal response options. In addition, the analyses of study 2 show that not all interviewers followed the instructions of reading the question with numbers very well. So also the manipulation was not always successful.

In conclusion, even minor interviewer deviations can influence data quality. Studying these deviations in pre-test interviews can assist in improving a standardized questionnaire.