The type of response scale may compromise the psychometric quality of a survey instrument even though its qualitative underpinning is excellent and the wording of the items is appropriate. A series of four experiments (one paper-and-pencil, two online, one eye-tracking experiment) were conducted to examine the functioning of response scales of different direction (agree-disagree versus disagree-agree). The effects of the direction of the response scale are exemplified by two different constructs, based on the Rasch measurement model. As a key result, the agree-disagree format performs better (in terms of fit and the unit of measurement) than the disagree-agree variant. Since a difference in the unit of measurement (a multiplicative bias) may lead to spurious mean differences, extreme caution has to be exercised when data sets based on different modes of collection or different response scales are merged. In search for a more detailed understanding of the differences found, an eye-tracking experiment was conducted. The eye-tracking experiment allowed to observe the answering process and to investigate the reading patterns of the respondents in a more direct way. In line with the results of the psychometric analysis, the two response formats also differed regarding reading patterns. For instance, fixation and observation length and counts of the defined areas of interest (e.g., scale-poles) were significantly higher for the disagree-agree format. These findings suggest the disagree-agree format implying more effort and difficulties for the respondents throughout the answering process.