An Experiential Account of Using Cognitive Interviewing Techniques Within a Repeated Measures Design in the Development of ONS Suite of Questionnaires Measuring the Service Sector in the UK (303572)

Keywords: establishment surveys, business surveys, pretesting, official statistics, cognitive interviewing techniques

The service sector dominates the UK economy and is the primary contributor to GDP. However this sector is notoriously difficult to measure with both conceptual and practical challenges. The pace of change within this sector means that developing and updating the ONS suite of official establishment surveys needs to demonstrate pace and flexibility in its approach. The aim of the presentation is to provide a practical account of the experience and the challenges encountered in using cognitive interviewing techniques within a repeated measure design and under project specific constraints while developing the ONS suite of official establishment surveys measuring the size and the structure of service sector in the UK. The work consisted of the redesign of industry specific paper questionnaires to a format suited for electronic data collection, while at the same time aligning the constructs intended for capture with international regulatory frameworks for establishment statistics. Aims discussed are the challenges encountered in conducting research while balancing a sound methodological approach with project specific constraints in a way that was conducive to complex changes to the questionnaires being quickly delivered. Cognitive interviewing techniques were used in pretesting to evaluate different questionnaires in a repeated measure design method while controlling for order effects. This approach facilitated the timely identification of issues and problems that the design team had not initially identified, both related to comprehension and response process issues but also structural problems such as routing (skip patterns) and layout. This in turn allowed the research team to gain insight into respondent’s interpretation of survey constructs within each layout option, aided the process of identifying areas of concern and informed changes to survey items in order to reduce survey error. As a result, a framework was developed to support similar research in the future.