Regression-Based Thinkaloud Analysis for Assessing the Validity of Survey Questions (303109)

Keywords: think alouds, regression, fear of crime, interpersonal trust

The use of think aloud protocols has a long tradition in survey research as a means of assessing the content validity of questionnaire items. This broad class of procedures generally requires respondents to vocalize their thought processes, either concurrently or retrospectively, as they read a question or questions. The recorded and transcribed ‘think alouds’ can then be used to assess whether the cognitive and affective responses elicited by the question(s) align with the concept the researcher wishes to measure. In general, these procedures have mostly (though not exclusively) been used in small group settings where the think aloud data is analyzed using ‘qualitative’ procedures that do not seek to make statistical inferences to the broader population from which respondents were drawn. In this paper, we describe a procedure---regression-based thinkaloud analysis---that captures think aloud data from a large random sample of respondents. The think aloud data is then coded to a frame that captures the conceptual content of the responses. In the analysis state, the coded responses are included as dummy variable predictors in a regression model, where the question that was used to elicit the think aloud responses is specified as the outcome. Individual-level demographic controls are specified so the estimated coefficients of the model can be used to interpret whether and to what extent the different cognitive frames identified in the think aloud data align with responses to the question. The technique can be extended to include split-ballot designs, where variants of the target question are randomized across groups to enable an assessment of how question content and response scale options may influence the cognitive frames elicited. Interactions between demographic variables and the think aloud dummies can also be specified to assess whether some frames are especially salient for particular population sub-groups. We illustrate the procedure using two example questions: interpersonal trust and fear of crime.