Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title Analyzing Cognitive Burden of Survey Questions with Paradata: A Web Survey Experiment
Year 2016
Access date 03.08.2016
Full text PDF (1,01MB)
Abstract
Relevance and Research Question: The measurement of attitudes, opinions, and behaviors with agree/disagree questions is a common and popular method in empirical social research. For instance, in the German General Social Survey, the Eurobarometer, and the ISSP this question type is frequently used. Fowler (1995), however, suggests that agree/disagree questions require an effortful and intricate cognitive information processing; he argues for the use of item-specific questions because they seem to be less burdensome. So far, this assumption lacks empirical evidence.
Methods and Data: In the current study, we examine cognitive burden of agree/disagree and item-specific questions in web surveys using paradata. Measuring response times makes it possible to examine the cognitive burden of different question types and provides insights into cognitive response processes. We used an innovative double-stage outlier correction that is based on the activity of the web survey while processing, followed by an outlier definition that is based on the distribution of the response times. Additionally, we captured computer mouse clicks to evaluate response times. We conducted a two group experiment that is based on an onomastic sampling approach. The first experimental group (n = 533) received eight agree/disagree questions on achievement motivation. The second group (n = 472) received eight similar item-specific questions on achievement motivation.
Results: Our findings suggest that the item-specific questions show, on average, significantly higher response times than their agree/disagree counterparts. The computer mouse clicks, however, show no significant differences between the two experimental groups. Hence, the question types do not seem to affect response times systematically.
Added Value: Altogether, it appears that item-specific questions contrary to the current state of research require a deeper cognitive information processing than agree/disagree questions.
Year of publication2016
Bibliographic typeConferences, workshops, tutorials, presentations
Print