A Compendium of Current Pretesting Methods (303124)

Keywords: pretesting, cognitive interviewing, usability testing, crowd sourcing

The purpose of this presentation is to review the traditional and newer methods currently available for survey pretesting and discuss considerations in choosing the ideal pretesting method or suite of methods when developing a survey.

While traditional pretesting methods such as expert review, cognitive interviews, and pilot testing have provided great insight into the errors to be addressed in surveys prior to fielding, they can be costly, time consuming, and limited in the range of perspectives gained. Additionally, they are not ideally suited for detecting specific concerns with web surveys, which have become increasingly popular in recent years. Web survey responses can be provided through a number of different devices such as computers, laptops, tablets, and smartphones. The use of these additional modes affects how respondents answer survey questions and consequently affects our ability to effectively pretest surveys offered in these modes. With changes in technology and communications in recent years, it is necessary to a re-evaluate the available pretesting options and new alternatives.

In the past few years, a number of emerging methods such as usability testing, eye-tracking, and crowdsourcing have been used to pretest questionnaires. These newer methods leverage advances in technology and communications to add additional insights to the pretesting process and/or complete pretesting in a more efficient manner. Yet, little is known about how effective these newer methods are in improving questionnaires, how they compare to standard pretesting methods such as cognitive interviewing, and whether these methods can complement existing methods by offering additional insights.

In this presentation, we provide a holistic analysis of both traditional and emerging questionnaire pretesting methodologies that survey researchers can use as a guide when planning for and budgeting the survey development process.