Thursday, November 10
Data Quality and Measurement Error
Thu, Nov 10, 3:30 PM - 4:55 PM
Regency Ballroom-Monroe
Cross National/Cultural Questionnaire Design: Strategies to Tackle Measurement Errors

A Model for the Cross-National Questionnaire Design Lifecycle (303106)

Rory Fitzgerald, European Social Survey, City University London 
*Diana Zavala-Rojas, European Social Survey, Universitat Pompeu Fabra 

Keywords: questionnaire design, cross-national survey research, comparability, total survey error

The total survey error (TSE) framework highlights the complex interactions between different design decisions at the various stages of the survey lifecycle, which can compromise the representativeness, reliability, and validity of the data. This complexity is magnified in cross-national studies, which often comprise a series of separate national surveys coordinated under a framework, with different levels of harmonization depending on the study.

Large-scale comparative projects such as the European Social Survey (ESS) study populations in more than 20 countries at a time. Their main objective is to produce comparable data across countries and, usually, this involves implementing an ask-the-same-question (ASQ) approach. Under the ASQ approach, a source questionnaire is developed to be suitable in one language and country whilst at the same time serving as a source instrument to produce multiple target versions. This dual role exemplifies the additional challenges for the cross-national researcher. They must not only deal with the known challenges of producing measurement instruments in one country, but also anticipate those of producing question versions for other countries.

In its early rounds, the ESS, like many cross-national surveys, developed its questionnaires through a single source language instrument in sequential stages: the source questionnaire was designed in British English, and pretesting beyond expert review was limited to country piloting toward the end of the process, just before the source questionnaire was finalized. Concerns about translation and comparability were therefore primarily raised and considered after the source questionnaire had been finalized. As a result, the process was too steeped in the source language and culture. However, alternatives such as simultaneous development of the questionnaire in multiple languages were considered too costly and logistically impossible to implement.