Breaking the Commandments of Survey Design: Cognitive Testing of a Commonly Used Scale of Subjective Religious Experiences (303623)

Keywords: reliability, data quality, cognitive testing, question evaluation

Despite well-documented survey design standards, some frequently used questions do not follow best practice, calling into question the quality of the collected data. To what extent do some commonly used survey questions about subjective experiences collect reliable data? To answer this question, questions from the Daily Spiritual Experience Scale (DSES)---a measure of subjective spiritual experiences used in over 200 published studies---were cognitively tested. Cognitive interviews were conducted with 25 Christian adults from the greater Boston area who varied in educational attainment, age, and race/ethnicity. Narrative analysis of the interview transcripts examined respondents’ comprehension of these questions and the cognitive processes behind their answers to these questions. Despite its popularity as a measure, the DSES fails to meet some basic question design standards, such as providing a time frame (rather than assuming regularity of experiences) and having mutually exclusive response categories. The results of the study suggest that the DSES collects data with substantial error and is unlikely to accurately measure respondents’ frequency of spiritual experiences as intended. Respondents with nearly identical experiences often selected very different responses from one another, as there were often multiple options that a respondent could reasonably choose from (e.g., if a respondent had an experience “many times a day” but only on “some days”). Additionally, respondents varied greatly in how they answered about spiritual experiences that they reported as “constant” or “always happening”---an issue potentially unique to questions asking about internal experiences of spirituality, but that were exacerbated by the scale’s failure to meet design standards. This study provides important insight into the quality of the data collected by this scale and perhaps other similarly structured scales, and enforces the importance of following the best practices of survey design.