Web Survey Paradata as a Tool for Evaluating Survey Design and Navigation (303347)

Temika Holland, U.S. Census Bureau 
Rachel Horwitz, U.S. Census Bureau 
*Courtney N Reiser, U.S. Census Bureau 
Jennifer Guarino Tancreto, U.S. Census Bureau 
Sarah Vetting, U.S. Census Bureau 
Adriana Hernandez Viver, U.S. Census Bureau 

Keywords: Paradata, Web Survey, Quality

Web survey paradata are an effective tool to help survey designers glean information about respondents’ experience completing a web survey. Paradata capture respondents’ actions with an associated timestamp as they progress through the survey. The insights provided by these data help survey administrators ensure that their designs minimize both respondent frustrations as well as measurement error. Web survey designers often rely on usability testing their instruments before implementing in a production environment. While usability testing is a great first step, pre-testing a web survey in the field provides greater opportunity to evaluate the instrument using paradata to show how users are interacting with the instrument in reality; specifically, what devices they are using, how long it takes them to complete, when and where are they breaking off, where they needed more context or help, etc.

In this paper, we illustrate the benefits of using web paradata to evaluate web survey design using the 2013 and 2015 National Survey of College Graduates (NSCG). We explore device usage, completion times, breakoff rates, error rates, rates of changed answers and use of auxiliary links to pinpoint areas of improvement in preparation for the next data collection cycle. We also study mobile device users to determine whether their experience differs from those accessing the desktop version of the instrument, in an effort to initiate improvements using a mobile-optimized version of the instrument. We then illustrate how issues identified through the paradata prompted changes to the instrument.