Relevance & Research Question: Rating scales with agree-disagree response options are among the most widely used question formats to ask for attitudes, opinions, or behaviors in Web surveys, especially because several items can be combined in a grid irrespective of whether the items measure the same or different constructs. Nevertheless, there is an ongoing debate on whether construct-specific (CS) scales are to be preferred to agree-disagree (AD) scales with regard to data quality and cognitive burden. Furthermore, due to respondents increasingly arriving at Web surveys via mobile devices, conventional grids are frequently being replaced by standalone (or item-by-item) question formats at least for mobile respondents. Thus, the question may arise why not make use of construct-specific scales, since standalone question formats are already on the rise in multi-device Web surveys.

Methods & Data: The present experiment was designed to gain a better understanding of how data quality and cognitive burden are affected by different kinds of scale formats. In a Web survey conducted among university applicants (n=4,477), a between-subjects design was implemented to examine three different scale formats in terms of an AD-grid, AD-standalone, and CS-standalone format. Moreover, scale direction was varied with response options being presented either in a positive-to-negative or reverse ordering. Several indicators of data quality and cognitive burden have been distinguished (e.g., response times, primacy effects, straightlining, and central tendency).

Results: Findings suggested that construct-specific scales prevent respondents from rushing through a series of items, thus encouraging a more thoughtful processing of the scale content without unnecessarily burdening the respondents. Moreover, construct-specific scales are less susceptible to variations in scale direction compared to agree-disagree scales. On the downside, respondents are more inclined to choose the middle category in construct-specific scales than in agree-disagree scales.

Added Value: The findings provide a better understanding of differences in the respondents’ processing of construct-specific scales compared to agree-disagree scales. Moreover, there is convincing evidence that construct-specific scales are a proper alternative to agree-disagree scales for both mobile and desktop respondents in multi-device Web surveys.