Approaches to Pretesting Electronic Questionnaires at Statistics Canada: Assessing Recent Approaches and Methodologies (303558)

Keywords: electronic questionnaire, EQ application, cognitive interviews, paper prototypes

Pretesting is widely acknowledged as the best way to assess whether a questionnaire may cause problems for respondents or interviewers. A primary goal of pretesting survey questionnaires is to improve data quality so measurements satisfy survey objectives and information needs.

In 2010, Statistics Canada introduced a new corporate project to implement electronic questionnaires (EQs) as the primary data collection instruments for most of its social and business surveys.

Statistics Canada has traditionally used cognitive interviews (among other techniques) as a principal pretesting methodology. For the past five years, cognitive interviews using fully functioning EQ applications loaded on laptop computers have been the main tool to pretest electronic questionnaires with respondents.

Due to demands on person-resources and production schedules, Statistics Canada has recently explored the merits and limitations of using alternate pretesting tools. Some studies have been carried out using paper prototype questionnaires and static PDF images of screen displays to assess and evaluate all elements of a questionnaire design including survey content, interaction design and visual design.

This paper will discuss and compare findings and observations from several questionnaire pretests that have used paper test instruments rather than fully functioning applications.