Thursday, November 10
Pretesting Methods
Data Quality and Measurement Error
Thu, Nov 10, 3:30 PM - 4:55 PM
Hibiscus A
Developments in the Design and Testing of Establishment Survey Data Collection Instruments

Cognitive Interviewing for Official Establishment Survey Questionnaires (303145)

Mojca Bavdaž, University of Ljubljana 
*Deirdre Giesen, Statistics Netherlands 
Ruth James, Office for National Statistics 
Jacqui Jones, Office for National Statistics 
Danna Lee Moore , Washington State University 
Paul Smith, University of Southampton 

Keywords: business surveys, pre-testing, official statistics

Cognitive interviewing is regarded as an important tool for identifying errors in the response process, causes of these errors, the resulting effect on responses provided to survey questions, and, ultimately, data quality. However, little is known about how cognitive interviewing is implemented in practice, especially in the context of establishment surveys. The purpose of this paper is to get insight into how statistical agencies use cognitive interviewing for the development, testing and evaluation of establishment survey questionnaires and how these approaches are evolving or adapting in line with technological developments.

The response process for establishment surveys differs in several ways from the typical response process in surveys for households and individuals. Whereas respondents in household surveys usually answer questions about themselves, based on information that they have instantly available or that can be retrieved from their memories, respondents in establishment surveys answer questions on behalf of an organization. The requested data for establishment surveys are usually not located in the memory of the respondent, but in business records. The quality and content of these records are important restrictions on the quality of the data that can be provided by respondents. Next, gaining access to the right respondents, who have the authority and capacity to provide the requested data, is an important issue in establishment surveys. Several persons in several roles (e.g., coordinator, data provider) may be involved in the completion of one establishment survey questionnaire. Often different collection modes are used for household and establishment surveys. For instance, there is a tendency for household surveys to be interviewer administered, whereas establishment surveys are typically self-completion. This has implications for the way that respondents interact with the collection instrument, and therefore the response process.

Establishment surveys in official statistics have some additional typical characteristics that affect the response process. The most prominent one is that reporting to official business surveys is usually mandatory. Another important characteristic of official business surveys is that they continue for many years and often the same businesses are sampled. Official statistics are typically interested in describing the state of the economy and for this purpose not all business respondents are equal. Large businesses have a crucial impact on population totals. As a result, these businesses are often (or always) sampled and intensively followed up in case of nonresponse. For some employees in these businesses, providing data for several official establishment surveys is a regular part of their job. Whereas for smaller businesses, being part of an official establishment survey sample is a rare event.

All these characteristics of the response process in official establishment surveys affect the possibilities and needs for designing cognitive interviewing studies. This paper focuses on how we are sampling and recruiting for cognitive interviews for establishment surveys, the types of interviews that are being conducted and how the data are analyzed and reported. It also investigates whether cognitive interviewing for establishment surveys takes on aspects of exploratory studies as well, including the exploration of data availability and respondent roles. More insight into how agencies are using cognitive interviewing will be helpful to see if certain practices can be considered standards. An overview of practices used will also help identify areas for further research to improve the testing and evaluating of questionnaires in establishment surveys.

First, we will give a short literature review of theories and recommended practices for the design of qualitative studies in general and cognitive interviewing studies in particular. We will compare this with the results of interviewing a selection of national statistical agencies on their practices regarding the use of cognitive interviewing for establishment surveys. Issues addressed will include how study goals influence study samples and recruitment designs; decisions on how and when to probe; just how in-depth to take questioning; and how data from cognitive interviews are documented, analyzed, and reported.