Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title The Adequacy of Outlier Definitions based on Response Time Distributions in Web Surveys: A Paradata Usability Study
Year 2016
Access date 29.04.2016
Presentation PDF (625KB)
Abstract
Relevance and Research Question: Web surveys are commonly used for data collection in empirical social research because they are cheaper, faster, and simpler to conduct than other survey modes. Furthermore, they enable researchers to capture a variety of additional data (so-called paradata) during the survey process such as response times. Measuring response times has by now a long tradition in social psychological research as well as survey research to investigate response behavior and response processes. One key problem, however, is the determination of appropriate thresholds to define outliers; to a certain degree researchers determine them arbitrarily. Until now, there is no scientific consensus with respect to the definition of outliers.

Methods and Data: In our study we developed an (innovative) two-stage outlier definition procedure for web surveys using paradata. This approach is based on the activity of the web survey while processing, accompanied by an outlier definition that is based on the distribution of the response times. Our web survey (n = 1899) is based on an onomastic sampling approach and contained individual questions as well as grid questions. Moreover, we tested different procedures for dealing with outliers based on the response time distributions.

Results: Our analyses show that common outlier definition procedures, which are based on the distributions of response times, provide insufficient results. This implies that they are frequently unable to capture respondents who leave the web survey for a short time period so that the response times are biased upwards. Particularly, this circumstance can be observed for grid questions.

Added Value: Altogether, our findings suggest that the two-stage outlier definition procedure is superior to common methods for dealing with outliers that are only based on response time distributions.
Year of publication2016
Bibliographic typeConferences, workshops, tutorials, presentations
Print

Web survey bibliography - Germany (361)

Page:
Page: