Using Pretest Results to Predict Survey Question Accuracy (303099)

Keywords: Pretesting Methods, Validity, Record Checks

Researchers use many methods to understand the quality of questions prior to fielding them. Previous research has compared methods according to the number and type of problems found, whether methods detect problematic questions in the field, whether methods successfully repair problems in the field, and whether methods predict the reliability of survey questions. But little is known about how the methods differ in identifying problems that affect the accuracy of the answers questions produce. Only a few studies have examined how the results from pretesting methods predict the validity of survey questions either through record check studies or model-based approaches to validity. This is likely because it is difficult to obtain records for most survey questions of interest to arrive at a clear understanding of the accuracy of survey questions. One approach to understanding the accuracy of survey questions through record check studies is to create a database of published record check studies. This approach was taken by previous authors to understand the characteristics of survey questions that may lead to inaccuracy. We take a similar approach in this study by examining how the results from question evaluation studies predict the accuracy of questions through record checks from the published literature. We compare the performance of five evaluation methods in predicting the accuracy of 51 questions for which validity estimates are available from record check studies. The five methods are Expert Review, the Questionnaire Appraisal System (QAS), the Question Understanding Aid (QUAID), the Survey Quality Predictor (SQP), and Cognitive Interviews. We find that the accuracy of survey questions based on the record check results is predicted by problems with socially desirable or sensitive content as identified by the testing methods. Furthermore, we find that a combination of expert review and cognitive interviews provides the best prediction of accuracy. We discuss the practical implications these findings have on the question evaluation process and the optimal choice of methods in this process.