Empirical Evidence for the Value of Usability Testing Surveys (303555)

Keywords: usability, usability testing, mobile, desktop

Cognitive pre-testing is often conducted prior to a survey launch to ensure questions and response options make sense to respondents. There is a growing body of literature toting the value of usability testing, which focuses on how design impacts the ability for respondents to correctly respond to survey items, yet it is often forgotten during pretesting stages. While the benefits have been argued, to date, there is no empirical evidence of the true value of usability testing of surveys. This study is the first to demonstrate the importance of usability testing in improving a survey before launch.

We created a self-administered survey and conducted iterative usability testing on desktop and mobile versions. In iterative testing, issues are discovered, changes are made, and the product is tested again, and this iterative process continues until optimal usability is achieved. In survey design, optimal usability is when the survey can accurately capture respondents’ selections. In this study, we conducted three rounds of usability testing, which led to design changes, mostly on the mobile version. We launched both the pre-usability tested version and the post-usability tested version to compare survey results.

In this talk, we will discuss the design issues that were discovered in usability testing, and we will present the data from both the pre-usability and the post-usability survey. We will show differences in breakoffs, item nonresponse, overall nonresponse, mean response distributions, and time to complete survey. Results will demonstrate the importance of conducting usability testing on a survey, across modes, prior to launch.