Combining Multiple Evaluation Methods: What Does it Mean When the Data Appear to Conflict? (303131)

Keywords: pretesting, cognitive interviewing, focus groups, piloting, evaluation, data triangulation, interpretation, validity

Over the past three decades, the use of a range of pretesting methods has increased. Methods can be used singularly or in combination and are typically used in the development of survey questions and data collection instruments. There have been relatively few studies that have compared pretesting methods, and such studies have focused on the reliability and validity of findings produced from a small number of different methods. The methods compared have been a subset of expert review, cognitive interviewing, behavior coding and respondent debriefing. While these earlier studies are useful, they do not cover some commonly used methods such as focus groups and piloting. In this paper, we consider:

•The level of consistency of findings from a wide range of pretesting method used in combination, including cognitive interviewing, focus groups, respondent debriefing, expert review, piloting, and administrative data •The kinds of data each method provides • What this tells us about how such methods can be combined and the data they produce interpreted, particularly when findings from different pretesting methods may appear at first contradictory

A framework is put forward that researchers can use to help them decide which pretesting methods to use.