Saturday, November 12
General
Sat, Nov 12, 10:30 AM - 10:55 AM
Promenade Upper
Saturday Poster Session, Part 1

Survey Focus: A New Procedure Defining Response Time Outliers in Web Surveys (303337)

*Jan Karem Höhne, University of Göttingen 
Dagmar Krebs, University of Gießen 
Stephan Schlosser, University of Göttingen 

Keywords: Empirical Social Research, Outlier Definition, Paradata, Web Surveys

Web surveys are commonly used for data collection in empirical social research because they are cheaper, faster, and simpler to conduct than other survey modes. Furthermore, they enable researchers to capture a variety of process data (so-called paradata) such as response times. Measuring response times has by now a long tradition in social psychological research as well as survey research to investigate response behavior and response processes. One key problem, however, is the determination of appropriate thresholds to define outliers; to a certain degree researchers determine them arbitrarily. Until now, there is no scientific consensus with respect to the definition of outliers. In our study we developed a new two-criterion outlier definition procedure for web surveys using paradata. This procedure is based on the acquisition of the activity of the web survey page while processing (called Survey Focus), followed by an outlier definition that is based on the distribution of the response times. Our web survey (n = 1899) is based on an onomastic sampling approach and contained individual questions as well as grid questions. Moreover, we tested different procedures for dealing with outliers based on the response time distributions. Our analyses show that common outlier definition procedures, which are based on the distributions of response times, provide insufficient results. This implies that they are frequently unable to capture respondents who leave the web survey for a short time; implying that the response times are biased upwards. This circumstance especially can be observed for grid questions. Moreover, our findings reveal that respondents who leave the web survey for a short time period produce significantly higher item nonresponse rates. Altogether, our findings suggest that the two-criterion outlier definition procedure is superior to common methods for dealing with outliers that are only based on response time distributions.