Using Calibration Training to Assess the Quality of Interviewer Performance

• Activity Number: 445
• Type: Contributed
• Date/Time: Tuesday, August 11, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Survey Research Methods Section
• Author(s): William Mockovak* and Morgan Earp and Alice Yu
• Companies: Bureau of Labor Statistics and Bureau of Labor Statistics and Bureau of Labor Statistics
• Keywords: Interviewer calibration ; Interviewer reliability ; Calibration training ; interviewer performance

Abstract

The Occupational Requirements Survey (ORS) is an establishment survey conducted by the Bureau of Labor Statistics (BLS) for the Social Security Administration (SSA). The survey collects information on vocational preparation and the cognitive and physical requirements of occupations in the U.S. economy, as well as the environmental conditions in which those occupations are performed. Calibration training is a type of refresher training that compares interviewer performance against predetermined standards to assess rating accuracy, inter-rater reliability, and other measures of performance. This paper will review the results of three separate calibration training sessions that focused on a data collector's ability to identify the presence or absence of physical demands and environmental conditions based on visual observation (assessed by watching job videos), assign Standard Occupational Classification (SOC) codes, and code Specific Vocational Preparation (SVP), which is a measure of the lapsed time required by a typical worker to reach average performance. Information obtained from these sessions was used to help evaluate training and mentoring programs, as well as to provide input into quality assurance procedures. However, the three calibration training sessions described in this paper generally showed minimal impact on performance measures used in the sessions.