The Role of Question Characteristics in Designing and Evaluating Survey Questions (303167)

Keywords: questionnaire design, question characteristics, question development, data quality, interviewer-respondent interaction

When developing questions for standardized measurement, question writers often focus on characteristics of the question (e.g., length or difficulty). Over time researchers have developed recommendations for writing questions that are formulated around these characteristics and that are based on results about how characteristics affect outcomes. However, while survey researchers know much about the effects of some question characteristics on data quality (e.g., Krosnick & Presser 2010; Schaeffer & Presser 2003), we argue that our field is still in the process of developing a comprehensive typology in which: question characteristics are cataloged and organized; effects on interviewers’ and respondents’ processing are understood; and effects on data quality are documented. Our presentation has several goals. First, we present a conceptual model for exploring effects of question characteristics on interviewers’ and respondents’ cognitive processing, indicators of processing, and data quality. Next, we provide an overview of some of the approaches used to conceptualize, measure, and code question characteristics. This overview includes an in-depth analysis of question characteristics provided by observational approaches versus experimental approaches. To further illustrate the observational approach, we examine the effects of characteristics (question, respondent, and interviewer) on indicators of processing difficulties for interviewers and respondents (i.e., question reading accuracy by interviewers and expressions of processing difficulty by respondents) in a case study. Finally, we put forth an agenda for future studies that includes: an examination of question characteristics and coding approaches with banks of questions with varied characteristics and an expansion and refinement of the approaches to coding characteristics.