Online Program Home
My Program

Abstract Details

Activity Number: 66 - Improving Data Collection: Challenges in Survey Practice
Type: Topic Contributed
Date/Time: Sunday, July 28, 2019 : 4:00 PM to 5:50 PM
Sponsor: Government Statistics Section
Abstract #306576
Title: Identifying Data Quality Outliers with Web Survey Response Times: Evaluation and Best Practices
Author(s): Y. Patrick Patrick Hsieh* and Joe Murphy
Companies: RTI International and RTI International
Keywords: Web survey; Paradata; Response time

As a means of quality control, survey paradata have been used to detect problematic interviews and responses. In particular, response time (RT) to a specific item, a particular question module, or the entire instrument are considered as informative measures for identifying performance outliers, such as speeders, draggers, or potentially falsified interviews. However, the quality assurance techniques used in the survey industry seems to be established based on practical experiences. There is scant research on determining cutoff thresholds for RT measures and the implications for detecting response outliers. In this study, we evaluated various behind-the-scenes techniques for using RT to detect response outliers. We found that using Tukey (1977) method to analyze normalized RT distribution to determine proper IQR cut-off threshold for response outlier detection may be the more robust technique for identifying RT outliers. We have replicated the technique in multiple studies and produced consistent execution for quality assurance. We further suggest how to examine additional satisicing patterns for quality assurance when RT outlier identification may not be effective.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2019 program