Validating the Results of an Establishment Survey of Occupational Requirements Using Direct Job Observations (303619)

Keywords: validation, establishment surveys, multi-method, evaluation

The Occupational Requirements Survey (ORS) is a new establishment survey conducted by the Bureau of Labor Statistics (BLS) for the Social Security Administration (SSA). It collects information on the vocational preparation and physical and cognitive requirements of occupations in the U.S. economy, as well as the environmental conditions in which those jobs are performed. ORS data are collected by BLS interviewers who conduct semi-structured interviews with an establishment respondent, typically human resource managers or occupational safety personnel.

Although the use of establishment-level respondents is pervasive in business surveys, these individuals may lack the knowledge necessary to provide accurate reports for some data elements. For example, early ORS interviewer debriefings indicated that respondents in corporate headquarters sometimes were unfamiliar with job requirements for occupations performed in a different corporate location (e.g., the incidence and duration of physical activities like lifting at-or-above the shoulder, stooping, etc.), resulting in higher levels of item missingness or estimation-based reporting strategies.

Given the potential drawbacks of using establishment respondents to collect ORS data, and as part of ongoing efforts to validate the ORS survey design, BLS conducted a test in 2015 to examine a collection method used in many smaller-scale job studies: direct observations of incumbents. The test demonstrated the feasibility of conducting job observations with ORS sample, and showed relatively high levels of agreement between observed and interview-collected ORS data. This paper will present the methods and key findings from this study, highlight methodological challenges and lesson learned, and discuss potential broader applications of this method for pre-testing, quality control, interviewer training, and validation in establishment surveys. We end with a discussion of plans for expanded ORS job observation testing.