Testing Forced Choice and Mark-All-That-Apply Question Formats to Measure Salient Social Identities in a Web Survey (303372)

Keywords: questionnaire design, web surveys, survey experiments, measurement error

Measuring social identities well---e.g., race/ethnicity, sexual orientation, gender identity---is important, especially given our diversifying society. But measuring multiple-response categorical variables like these can be difficult, particularly in self-administered modes. Both commonly used question formats have drawbacks. Mark-all-that-apply (MATA) reduces respondent burden, but generates data that conflate intentionally unendorsed categories with item nonresponse. Forced choice---changing each option into a yes/no question---potentially increases response validity but also increases respondent burden, yielding breakoffs, partial interviews, and unit nonresponse. In both formats, the longer the list of options, the higher the burden on the respondent, and the greater the risk of satisficing and other measurement errors that can result in poorer data quality.

In this study, we test the effectiveness of these formats using each to measure a social identity of potentially high salience. 2708 respondents from a nonprobability web panel, stratified by sexual orientation (half LGBQ, half heterosexual/straight) were asked about their gender identity and sexual orientation in one of six (2x3 factorial) questionnaire designs: (1a) MATA or (1b) forced choice, presented in (2a) one question, (2b) two questions on one page, or (2c) two questions on two pages.

Given that high question salience can increase respondent motivation to conscientiously answer the question, we investigate data quality between strata. Do respondents for whom the measured identity is of high salience (LGBQ) overcome the poor design of MATA? And what are the measurement properties heterosexual/straight respondents?

Findings suggest that MATA works adequately for all respondents, as long as both gender identity and sexual orientation are asked in two questions on one page. While forced choice works slightly better, it too can encounter the missing data problems intrinsic to MATA.