Should We Agree to Disagree? Further Considerations About Indirect Rating Scales (303618)

Keywords: survey questions, indirect rating scales, direct rating scales, agree-disagree scales, survey design

Likert scales in the Agree-Disagree form are a popular style of survey questions. However, experts in the field of survey design have criticized the style, citing that the format compromises data quality and that direct rating styles yield higher quality data. Among other problematic characteristics of Agree-Disagree questions, it is hard to know if or what meaningful differences exist between “Strongly/Completely Agree” and “Somewhat Agree,” or between “Somewhat Agree” and “Somewhat Disagree.” As part of a larger study using cognitive interviewing to evaluate survey questions, thirteen respondents answered a short series of opinion questions about national security. Two of the questions were asked twice with parallel question wording, but with different response options. The first version of the questions provided four response options (Completely Agree, Somewhat Agree, Somewhat Disagree, and Completely Disagree), while the other asked respondents to provide a numerical value on their level of agreement, with options ranging from 0 to 10 (with 0 meaning “Completely Disagree” and 10 meaning “Completely Agree.”) By comparing respondents’ numerical rating of agreement with their selection on the ordinal scale of agreement, we better understand the magnitude to which one needs to agree or disagree before choosing an ordinal Agree-Disagree category. In addition, a third question was asked on the same topic, using question specific wording and a direct rating scale (rather than an Agree-Disagree form). The answers to this direct rating question were also compared with the answers to Agree-Disagree forms. Patterns in the qualitative data gathered from the cognitive probes will also be discussed, including instances in which respondents requested to change their original answers during the cognitive probing, and instances in which respondents’ narratives surrounding their answer choices were contradictory to their question responses.