The Questionnaire Testing Bento Box: Results from Multi-Method Questionnaire Testing for the 2017 Census of Agriculture (303126)

Keywords: cognitive interview, field test, multi-method

A traditional Japanese bento box meal is constructed according to principles of balance---separate compartments with different colors, flavors, and cooking methods provide an optimum eating experience. As in the bento box, questionnaire testing methods can be combined to provide a balanced questionnaire evaluation.

The quinquennial Census of Agriculture (COA) is the largest data collection conducted by the U.S. Department of Agriculture’s National Agricultural Statistics Service. Farm size, type, production, economics, and demographics of farm operators are collected with a self-administered form, mailed to ~ 3 million potential farms. Similar to the 2012 COA, NASS is using multi-method questionnaire testing for the 2017 COA.

Initial activities included internal expert reviews by NASS subject matter and production staff. In addition, an external expert panel provided recommendations regarding the measurement of women and beginning farmers. Item edit and imputation rates in the 2012 COA were reviewed. The 2012 telephone help calls for the form were also reviewed. These initial reviews and proposed new content resulted in initial questionnaire drafts. Multiple rounds of cognitive interviews were then conducted focusing on new material and form changes.

In early 2016, a field test with ~30,000 records was conducted. Data from this test will be analyzed to evaluate the forms and identify any remaining problem areas. The test will also serve to test census processing systems. Respondents in the field test with suspected reporting errors will also be contacted in follow-up cognitive interviews.

All of our questionnaire bento box elements will be combined to make final recommendations for the 2017 census questionnaires. Each of the bento box compartments and the complementary information it provides will be discussed with selected results as illustration. Lessons learned to help improve the process for the 2022 COA will also be discussed.