The Interplay Between Survey Research and Psychometrics, with a Focus on Validity Theory (303143)

Keywords: validity, validation, survey design, questionnaire design, psychometrics

The concept of validity is one of the most influential concepts in survey research because considerations about its scope and nature influence everything from survey and questionnaire design to implementation as well as the use and reporting of results. Curiously, quite independent lines of thought and practice have evolved in survey and questionnaire design, and psychometric theory. We will explore the interplay between recent developments in psychometric theory (e.g., Zumbo’s DLD framework, as well as developments in cognitive interviewing and validity theory) and questionnaire and survey design to draw conclusions about data quality and inferential quality---all with an eye toward bridging concepts and practices. We will see that survey research and psychometrics have approached matters quite differently. Survey research has put data quality (e.g., estimates of error variance and sampling variability) at the forefront, whereas contemporary psychometric validity theory, by and large, focuses on characteristics of the inferences that are drawn about phenomena such as attitudes, self-reports, or opinions by human agents and the actions that result from these inferences. We will see how both disciplines’ claims about data or inferential quality, however, are always a matter of degree and not absolutes.