Necessary but Insufficient: Why Measurement Invariance Tests Need Online Probing as a Complementary Tool (Winner of NCHS Monroe Sirken Award) (303308)

Keywords: Web Probing, MGCFA, Cross-National, Mixed Methods

Over the last decades, a tremendous increase has occurred in cross-national data production in social science research. Before drawing substantive conclusions that are based on cross-national data, it is necessary to assess the comparability of data. Two main approaches to the assessment of comparability can be distinguished—those using quantitative methods and those using qualitative methods. Multi-group confirmatory factor analysis remains the predominant quantitative approach that is applied by the majority of substantive researchers. However, if the quantitative test fails to establish cross-national comparability, this approach struggles to explain the existing noninvariance. Online probing is an innovative qualitative method that has been developed recently. Online probing primarily seeks to uncover the causes for the lack of comparability of items. The drawbacks of this method are the necessity of collecting data and the work-intensive analysis, which limit the analysis to a small set of countries. Since both methods share complementary strength and weaknesses, much can be learned through a combined approach of both perspectives.

This presentation closes a research gap by simultaneously applying MGCFA and OP to assess the comparability of the constructs of constructive patriotism and nationalism for five countries (Germany, Great Britain, Mexico, Spain, and the U.S.). Using the 2013 ISSP Module on National Identity and data from a web survey conducted with 2,685 respondents in May 2014, I will demonstrate how the open-ended responses from OP can explain why scalar measurement invariance tests in MGCFA failed for these constructs. Finally, optimal research strategies that combine the two methods’ respective strengths will be presented.