Improving the Question Appraisal System (QAS): Moving Further Away from Black Magic and Black Boxes
Melinda Scott
ICF
Matt Jans
ICF
Ashley Schaad
InMoment
Questionnaire design can be the least transparent survey lifecycle phase, viewed by some researchers as "black magic" accomplished only by highly-trained survey methodologists. One common approach involves a trusted designer who reviews and revises the instrument with or without justifying their edits. The Questionnaire Appraisal System (QAS-99) was developed to a) make this process replicable and transparent, and b) allow review and revision by staff with less training and experience. The QAS-99 includes eight review steps focusing on question characteristics that lead to difficulty answering accurately. QAS-04 further improved upon review by adding steps covering translatability, cross-cultural assumptions, and cross-question issues. This paper presents new QAS developments which a) separate the individual question review and total instrument review into two assessments, and b) add a review step focused on encoding. Results show that the QAS can be used by survey researchers with little or no questionnaire design experience, and that there is a wide range of reliability across cognitive problems rated by the tool. Avenues for further QAS development are discussed.