A Framework for Making Decisions About Question Evaluation Methods (303100)

Keywords: question evaluation, expert review, laboratory methods, field methods, best practices

There are a variety of methods available to researchers and practitioners to develop and evaluate survey questions. However, this diversity of methods can make it difficult for practitioners to decide which methods to use in specific situations. This paper begins with a brief overview of the major methods available for questionnaire development and testing, including different types of expert, laboratory, and field-based methods for evaluating survey questions. The focus of the paper is to present a framework for practitioners to make decisions about question evaluation methods. To achieve that goal, the paper first presents a set of factors to consider when making decisions about which methods to use. Many factors influence the choice of methods used for a particular project. For example, the budget and time available for questionnaire development and testing are often the major factors in determining which methods can be used. Expert methods (which require no data collection) are the least expensive and require the least time; field-based methods are the most costly and most time consuming. The methods also differ in the type of expertise required for implementation. For example, some methods require qualitative skills, whereas others are more quantitative in nature. We identify these decision factors based on best practices and the results from some recent studies that have compared how the results from the different methods are similar or different. Second, the paper presents a decision tree that can be used by practitioners that can help to identify appropriate methods to use for their question evaluation project.