Purposefully Mobile: Experimentally Assessing Device Effects in an Online Survey

• Activity Number: 494
• Type: Contributed
• Date/Time: Wednesday, August 12, 2015 : 8:30 AM to 10:20 AM
• Sponsor: Survey Research Methods Section
• Author(s): Frances M. Barlas* and Randall K. Thomas and Patricia Graham
• Companies: GfK Custom Research and GfK Custom Research and GfK Custom Research
• Keywords: mobile ; questionnaire design ; online survey ; smartphone ; tablet ; device effects

Abstract

The recent growth in the number of participants who take surveys on mobile devices can be seen as an opportunity to increase coverage and reduce non-response, but it is essential that online surveys be designed with mobile display in mind to improve data quality and minimize device-specific effects. We conducted a study using GfK's KnowledgePanel, the largest online probability-based panel in the US to inform recommendations for a mobile-first study design. Rather than rely on 'accidental mobile' participants with their attendant self-selection bias, participants were randomly assigned to complete our survey on one of three devices (desktop/laptop, smartphone, or tablet). We had 4,555 completed surveys with an average completion time of about 18 minutes. We will present the factors that affect completion and substantive response differences, including device, survey design, and demographics, behaviors, and attitudes of participants. We will also discuss observed differences in response that appear to occur because of device used and not due to sample differences as well as provide recommendations for designing online surveys to accommodate mobile respondents.