Challenges to Developing New Survey Questions: When Cultural Norms Run Counter to Survey Questions (303130)

Keywords: pre-testing, questionnaire development, cognitive interview, content test

The Census of Agriculture (COA) is the main source of information on characteristics of farmers and ranchers. Since the publication of the 2012 COA, the National Agricultural Statistics Service (NASS) has received feedback from the agricultural and government sectors that the roles of women are not being adequately measured in the COA.

In April 2015, an expert panel was convened to re-evaluate how NASS measures the contribution of women and to provide recommendations for improvements to the COA. The panel identified two significant issues that NASS must address in the 2017 COA: the evolution of farm structure and cultural bias in which the family patriarch is viewed as the principal operator even if this individual has limited or no day-to-day involvement in farming.

Following the panel, NASS tested a new operator characteristics section using a multi-method approach as part of the 2015 Census Content Test. Iterative rounds of cognitive interviewing were conducted. We found that some respondents were still underreporting the role of women. Respondents cited various reasons for why they did not report women decision makers such as the women on the farm contribute to decisions but they do not make the final decisions (“We always argue her down”), women were involved in fewer decisions than men (“She just does the bookkeeping”), and because of religious beliefs (“The bible says he’s in charge”). These examples show that some cultural bias still affects the response process.

In early 2016, a Content Test Mailout was conducted using a sample of approximately 29,000 farming operations. Analysis will examine how responses (namely reporting of women farmers) changed on matched records from the 2012 COA. Additional analyses will be performed to assess the validity and reliability of the new questions.

In this paper, we will present results from our mixed-methods study and discuss the challenges of developing survey questions that challenge social norms.