Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title How the Order of Response Options in a Running Tally Can Affect Online Survey Estimates.
Year 2011
Access date 30.07.2011
Full text

PDF (184 kb)


In the design of online surveys, running tallies or constant sums are used to help respondents sum up the allocation of amounts so that the total sums to 100%. We hypothesized that for time allocation, the order of the presentation of the time categories could make a difference in the distribution of reported time spent. We expected primacy effects, with the firstpresented time category having a higher allocation of time than the later-presented options. Two independent experiments were conducted, one on a sample of Google online advertisers, and one on a general population adult sample from KnowledgePanelŸ. In the Google experiment, advertisers were asked to estimate percent of time to complete an online task consisting of 3 subtasks. The order of the subtasks was randomized. Consistent with our hypothesis, we found primacy effects -- the subtask presented in position 1 obtained a higher mean allocated time than when presented in later positions. In the KnoweldgePanel experiment, respondents were asked to provide running tallies of the percentage of television they typically watch during the morning, afternoon, and evening (separately for weekdays and weekends). As before, the order of the subtasks was rotated. Primacy effects were detected again, however differences by position were small and not statistically significant. Because time spent watching TV is a regular activity, viewing patterns are more likely to be encoded or ingrained in memory, and more likely to be reported reliably, with responses less susceptible to order effects. In the second experiment, we also tested the restriction that running tallies must sum to 100%. Roughly 25% of respondents were prompted for out-of-range sums on the first tally (weekday) and 5% were prompted on the second tally (weekend). This drop-off in prompting indicates that respondents quickly learned what is required to complete a given task.


Access/Direct link

Conference Homepage (Abstract)


Year of publication2011
Bibliographic typeConferences, workshops, tutorials, presentations

Web survey bibliography - Callegaro, M. (70)

  • 1
  • 2
  • 1
  • 2