Web Survey Bibliography
Previous research shows that changes in number-label association can affect respondents’ answers. In our first study, conducted with members of KnowledgePanel®, we manipulated the order of presentation of a fully labeled five point satisfaction scale (very satisfied to very unsatisfied); the association of numbers from 1 to 5 to each scale point; and we also reversed the number order (from 5 to 1). This manipulation created six experimental conditions. For each condition, we presented seven different items on separate screens. The completion rate for this study, done in April 2008, was 71%.
In the second study we used only a polar point satisfaction scale and manipulated the order of the scale as well the number-label associations (1 to 5 or 5 to 1). This created 4 experimental conditions to which KnowledgePanel members were randomly assigned during a survey conducted in August of 2008 with a completion rate of 72%. For each condition, we presented seven different items on separate screens.
In the first study we investigated the differences between each version, focusing specifically on the following comparisons: response options with numbers associated vs. options without numbers; order of the response option presentation; and order of the number association. Results from experiment 1 suggest that (1) reversing the order of presentation makes a significant difference resulting in primacy effects and in an increase of time latency when the scale is shown starting from “Very unsatisfied”. (2) Changing the number-label association does not make much of a difference; it appears that for a fully labeled scale the label is almost overriding the number. (3) Respondents are, however, slightly confused when the number-label association does not go in the expected direction.
In the second study we compared the 4 experimental groups, first between (same order of presentation but reversing the number-label association), and then within (same number-label association but reversing the order of presentation of the scale). We did not find many significant differences among groups, thus leading us to conclude that for a polar point scale the number-label association has less of an impact than for a fully labeled scale.
Homepage (abstract)/(full text)
Web Survey Bibliography - Wells, T. (9)
- A Direct Comparison of Mobile Versus Online Survey Modes; 2012; Wells, T., Bailey, J., Link, M. W.
- Catch Them When You Can: Speeders and Their Role in Online Data Quality; 2011; Gutierrez, C., Wells, T., Rao, K., Kurzynski, D.
- How the Order of Response Options in a Running Tally Can Affect Online Survey Estimates.; 2011; Callegaro, M., DiSogra, C., Wells, T.
- Representing Seniors in an Online National Probability Panel Survey: Measuring Technology Attitudes...; 2010; Peugh, J., Mansfield, W., Wells, T., Semans, K.
- Differences in Length of Survey Administration between Spanish-Language and English- Language Survey...; 2010; Wells, T., Vidalon, M., DiSogra, C.
- The Challenge and Importance of Including Spanish- Dominant Latinos in Online Panel Studies Addressing...; 2009; DiSogra, C., Wells, T., Torres, J.
- The Challenge and Importance of Including Spanish-Dominant Latinos in an Online Panel; 2009; Dennis, J. M., Wells, T., Torres, J.
- Effects of Pre-coding Response Options for Five Point Satisfaction Scale in Web Surveys; 2008; Callegaro, M., Wells, T., Kruse, Y.
- Do online respondents go the extra mile and take on inconvenient tasks?; 2008; Callegaro, M., Wells, T.