Saturday, November 12
Questionnaire Design
Sat, Nov 12, 4:00 PM - 5:25 PM
Hibiscus A
Design Trade-Offs

Survey Package Design and Response Rates in an Outdoor Recreation Study (303151)

*Jessica Behm, Westat 
Cynthia Helba, Westat 
W. Sherman Edwards, Westat 
Mina Miller, Westat 
Regina Yudd, Westat 

Keywords: mail questionnaire, survey package design

The Association of Fish and Wildlife Agencies (AFWA) awarded Rockville Institute a grant to convert a survey of wildlife-associated recreation from telephone and in-person administration to mail. We chose a traditional design for the mail survey package: a simple graphic cover with color, AFWA’s logo, and a letter from AFWA inside the survey booklet. We pretested this screener with a national sample of addresses and sent a reminder post card and a second screener mailing, half by FedEx.

The response rate for the pretest screener was a disappointing 17%, especially since another recent mail pretest had yielded 50% response. The design of that questionnaire was informed by testing of different cover “looks.” The selected design was “official”-looking, with black-and-white photographs and an emphasis on the government sponsor.

We then designed a second screener pretest. The instrument was in black and white, with two black and white cover designs: one with wildlife-related photographs and one with a line drawing of an outdoor scene. The survey package had a separate letter, signed by an official of the agency in the sampled state with AFWA membership, and included that agency’s logo. The results of this pretest were encouraging and the “official” design was selected for the main study, which included a split-ballot experiment in four states testing the agency logo and letter against the AFWA logo and letter. For the other states, we used the AFWA logo and letter.

This paper compares response rates for the two pretest screeners and the main study screener, including the experiment, to assess whether changes to the package design improved response. It also assesses “avidity bias” in the responses to the two screeners and the main study, i.e., the likelihood that those most interested in the topic are more likely to respond, biasing estimates of total participation and associated costs.