The Impact of Mobile Web Design on Survey Results (303148)

Keywords: mobile questionnaire design, device agnostic, nonresponse, break-off rates, forced vs. voluntary response, nonsubstantial response alternatives

To adapt to a mobile world, we need to find new, creative ways of reaching and engaging respondents, especially those below 40. They expect well-designed surveys and questions to respond to when and where it suits them on a device of their own choice. They also expect it will not take much of their time. We need to make shorter, smarter, device-agnostic surveys. This requires changing the way we think about and design questions and questionnaires.

Smartphones have the smallest screens, and if a question performs well on a smartphone, it will likely work well on larger screens. Research shows it takes longer to respond to a survey on smartphones than on larger screens; that break-off rates are higher in longer surveys; and that grids, large scales, long answer lists, radio buttons, and check boxes do not work well on small screens. Whether you should be able to skip questions and providing a “don’t know” (“DK”) response alternative are other important design issues. Most online sample providers do not permit skipping questions or provide a “DK” alternative.

In this paper, we will present results from a recent experiment in which we tested different elements of survey design. We designed eight variations of the same questionnaire. The sample was randomized into eight parallel groups with equal probability to be selected for each group. We used GallupPanelet, a high-quality, probability-based panel.

The hypotheses we wanted to test were whether a mobile-friendly design gives better response quality, whether voluntary response results in fewer break-offs than mandatory response and whether a “DK” alternative results in fewer break-offs, especially in surveys with forced response.

In this paper, we will analyze the different samples and effects of the different designs on response rates, unit nonresponse, item nonresponse, and break-off rates. We will conduct a comparative analysis of relevant variables and analyze feedback given by the respondents.