Recent studies have indicated that members of Internet panel surveys now complete surveys on PCs, but also on subnotebooks, tablets or smartphones (Bosnjak et al 2013). This means that web surveys are displayed on very small and very large screens, and are completed with keyboard, mouse or touchscreen
The fact that Internet surveys can now be completed with so many different devices may have implications for measurement. There have been only a few studies studying the effects on measurement error between different devices. Most studies have concentrated on comparing mobile phones to desktop PCs and have found no clear differences between them with regards to measurement error. Because respondents always self-select into particular devices, it is very difficult to separate selection effects from measurement effects of the device however.
This paper aims to fill this gap by using longitudinal data from a large Dutch probability-based panel survey (LISS) to analyse over time 1) whether the same respondents use different devices to complete Internet surveys and 2) assess whether a device switch over time affects survey measurement error. To this end, we coded the User Agent Strings of 6 waves of data, and categorized respondents according to the devices used over time. We identified several groups of ‘stable’-device users (e.g. “always PC”) and ‘switchers’ (e.g. PC -> tablet -> tablet -> PC).
We study the measurement properties of each of the devices using a within-subjects design. We look at 1) the number of Don’t knows and item missing 2) interview duration 3) straightlining 4) the length of open answers 5) primacy effect in check-all-that-apply questions and 6) how the survey is evaluated at each wave
We conclude with a discussion of the effects our findings have for the design of web surveys.