Online Program

Surveying hard-to-reach research population groups: A mixed methods approach to questionnaire development and evaluation

*Gordon B Willis, National Cancer Institute 
Bryce Reeve, University of North Carolina 
Salma Shariff-Marco, Cancer Prevention Institute of California 
Nancy Breen, National Cancer Institute 
Kerry Levin, Westat 
Martha Stapleton, Westat 
Gilbert Gee, UCLA 
David Grant, UCLA Center for Health Policy Research 
Ninez Ponce, UCLA 
Timothy Patrick Johnson, University of Illinois at Chicago 
Margarita Alegria, Harvard Medical School 
Nancy Krieger, Harvard School of Public Health 
Hope Landrine, East Carolina University 
Vickie Mays, UCLA 
David Takeuchi, University of Washington 
David Williams, Harvard School of Public Health 
Benmei Liu, National Cancer Institute 
E. Richard Brown, UCLA 

Keywords: Questionnaire design, cross-cultural, comparability, multicultural, multilingual

An often overlooked aspect of the concept of "Hard to Reach" is the development of questionnaires that function effectively across culturally diverse population groups. Challenges include: (a) asking survey questions that adequately capture respondents’ experiences; (b) using terminology that is well-understood; and (c) achieving comparability across survey translations. Although these issues are not unique to H2R populations, it is especially important to address them when surveying these populations, to ensure measurement comparability across groups. We applied a mixed-methods design to pretest questionnaire items administered in two field tests of the Discrimination Module within the California Health Interview Survey (N = 12,144). The first test focused on contrasts between six racial/ethnic groups (African-Americans/Blacks, American Indian/Alaska Natives, Asian Americans, Latinos, Multiracial respondents and Non-Latino Whites). The second test focused on six language groups (English, Spanish, Korean, Vietnamese, Cantonese, and Mandarin). We applied both qualitative and quantitative methods: cognitive interviewing, behavior coding (with multi-level statistical modeling), and psychometric testing (including assessment of Differential Item Functioning).

We will present examples from our mixed methods approach that demonstrate how threats to item validity across groups were identified and resolved. Results indicated that: (a) many problems crossed cultural boundaries (e.g., the terms ‘courtesy’ and ‘respect’ were generally perceived as redundant); (b) response categories such as ‘always’ and ‘sometimes’ tended not to align across languages, requiring re-translation; and (c) questions presented problems when they made assumptions that did not apply to certain groups (e.g., questions about lifetime discrimination created problems for immigrants who had lived within multiple socio-cultural contexts). We conclude by encouraging the use of a mixed-method approach for developing instruments intended for hard-to-reach populations, and for ensuring measurement comparability of items across a range of cultural groups and languages.