Optimizing Questionnaire Design in Cross-National and Cross-Cultural Surveys (303092)

Keywords: comparative, cross-national, questionnaire design

Questionnaire development in cross-national and cross-cultural surveys starts with the same challenges that exist for single monolingual and monocultural surveys. The individual measures need to be reliable and valid and need to function well among the target population (e.g., understandable, answerable, consistently understood across respondents). Furthermore, when multi-item measures are to be used, the scales must measure well the concepts of interest (e.g., high Cronbach’s alpha, strong and appropriate factor loadings). But then for cross-national/cultural surveys, the process becomes more complicated and challenging. The individual measures and scales need to achieve a high level of functional equivalence across surveys and minimize comparison error. Achieving this is not easy. First, one needs to optimize wording comparability by applying rigorous translation procedures such as the TRAPD model (Translation, Review, Adjudication, Pretesting, and Documentation) and the analysis of results for differences due to measurement variation resulting from errant or at least suboptimal translations. Second, one needs to consider whether there are structural or cultural differences (besides language) that undermine the comparability of items. Third, when multi-item scales are involved, one needs to test for equivalence (e.g., by using IRT or CFA procedures). Then if, as is often the case, equivalence is not established, one needs to identify the source of the nonequivalence. There are numerous possible reasons for this such as a) poorly operating scales, even within countries; 2) poor translations; 3) structurally or culturally based differences in specific items; or 4) true and meaningful differences in the configuration of values and attitudes related to a concept due to substantive variation in how societies assess the concept. The source for the nonequivalency must be ascertained, since the steps that need to be taken to deal with it depends on its source and nature.

Operational and methodological techniques have been developed to deal with this challenge, and these will be discussed. These include a) best practices to carry out and evaluate translations; b) design-to-archiving metadata protocols to maximize the careful development and documentation of the surveys (e.g., the Questionnaire Design Documentation Tool and Translation Management Tool); c) coordination across data collection to fully implement and enforce input harmonization from the conceptual level down to the nitty-gritty of coding, cleaning, and other details of surveys; d) items development aids such as SPQ2 and QUAID; and e) scale evaluation procedures such as IRT, CFA, and Bayesian approaches.

Detailed examples and specific innovative approaches used by major international surveys will be discussed. In particular, the European Social Survey; International Social Survey Program; and Survey of Health, Ageing, and Retirement in Europe will be used.