Relevance & Research Question:

The increasingly mixed technical devices respondents use to answer online surveys cause concern among survey researchers. Despite efforts to adjust online surveys to smartphones knowledge is still limited concerning how different response scales work on different devices. The main concern is that respondents who use a smartphone might treat response scales different than PC or laptop users. This paper analyzes response order effects (e.g. primacy effects) and uses large sample sizes that enable us to analyze whether these effects are conditional on scale length, question orientation (horizontal vs vertical) and the type of response device participants are using.

Methods & Data:

This paper reports four experiments where response option order, scale visual orientation (horizontal or vertical) also scale length are varied randomly. These experiments examine effects for dichotomous scales, 3-point scales, 5-point scales, 7-point scales and 11-point scales for political attitude questions. The data was collected by the Laboratory of Opinion Research (LORE) at the University of Gothenburg, Sweden, from their online panel (The Swedish Citizen Panel) between May 2014 and June 2015.

Results:

Results indicate that vertical response scales suffer more from primacy effects than horizontal response scales, and that this is even more so for smartphone users. Moreover, we find an effect of scale orientation in itself apart from response order effects: vertically oriented scales produce more responses in the beginning of the scale (topmost) than horizontally oriented scales do (leftmost). This means that comparability between smartphone and PC data is threatened when questions are not shown in the same way to respondents on different devices. However, scale length clearly matters in that these effects are substantially larger when scales are 5 points or longer.

Added Value:

This paper demonstrates the importance of question orientation (horizontal vs vertical) and that these should be identical on smartphones and PCs to ensure comparability. Since each of the four experiments contain at least 10,000 participants and the share of smartphone users is substantial (between 20 and 30 percent) these can be analyzed with sufficient statistical power.