Survey authors commonly have respondents rate or rank a series of items along some dimension of judgment. Alwin and Krosnick (1985) indicated that the different tasks leads to significantly different latent structures among the variables.
Method Respondents:
1882 respondents participated, randomly drawn from the Harris Poll Online panel.
Design:
Target assigned (more often visited or less often visited grocery store) , Number of elements to evaluate (5 versus 10)
Evaluative task:
Absolute rating of quality , Comparative rating of quality , Importance rating , Likelihood of influence , Quality Ranking , Importance Ranking
Procedure:
1. Asked about grocery stores they visited.
2. Assigned 1 store to rate
3. Rated familiarity with store and each element
4. Rated criteria (e.g. overall evaluation)
5. Evaluated store on elements (e.g. store’s price of products).
6. Asked 2 questions on task difficulty and accuracy.
Results:
Importance ratings paralleled importance rankings in terms of order. Rating means did not change between the 5 and 10 element conditions, but there were significant shifts for the ranking means. The Absolute Rating of Quality and Comparative Rating of Quality groups had significantly higher average correlations with the criteria (.48 and .45, respectively) than with the Absolute Ranking of Quality group (the highest average correlation for a single element with was .13). Respondents with 10 elements perceived that the task was more difficult and they were less accurate than those with 5 elements. Respondents assigned rating tasks perceived the task as easier and they felt more accurate doing it than those assigned the ranking task.
Discussion:
Ranks obtained seemed to be dependent on the presence or absence of other elements, so the selection of the other elements seems a critical, yet understudied, area.