Web Survey Bibliography
Title A Comparison of Ranking versus Rating in Online Surveys
Author Thomas, R. K.
Source RC33 6th International Conference on Social Science Methodology: Recent Developments and Applications in Social Research Methodology, 2004
Access date 10.09.2004
Abstract Traditionally, rating and ranking measurement operations each have had their proponents. Some criticisms of rating tasks have been that they are susceptible to end-piling (giving high ratings to all targets), often have a high degree of multi-collinearity, and often fail to get people to sufficiently differentiate between factors affecting their decision-making. Some criticisms of ranking tasks have been that they: 1) force dependence in responses which reduces the power of statistics that can be applied to this kind of data, 2) the subjective meanings of a given rank can vary substantially among individuals, making comparisons between individuals, and 3) do not provide distance information between the responses (i.e., is the difference between number 1 rank and number 2 rank equal to the distance between 2 and 3?). Thomas and Shaeffer (2003) and Thomas, Johnson, and Behnke (2004) have reported that, when compared to ranking tasks, rating tasks were less burdensome on respondents and more valid in terms of relating to key criteria in areas as diverse as grocery store attribute evaluations to political issue support. This study sought to extend our understanding of the differences in rating and ranking tasks in yet another area – body image. In addition, we conducted a recontact study to assess the reliability of rankings versus ratings. Method. In our first wave, over 10,000 respondents from the U.K. participated in an online survey. They were asked about their overall perceptions of their appearance and their satisfaction with the appearance of 4 aspects of their body (face, upper body, midsection, lower body). They were then randomly assigned to either rate or rank each of the 4 aspects in terms of physical attractiveness to others. Approximately 30 days after the first wave, we recontacted a subset of respondents to invite them to participate in another survey containing the same items measuring body image, of which 2832 respondents completed the second wave. Results. First, as has been obtained in the other studies comparing rating and ranking methods, we found that the order of means of the elements did not vary substantially whether they were rated or ranked. Second, we again replicated the greater validity of the rating scales (with an average validity twice that of ranking). Third, we found that rating tasks took far less time to complete. Fourth, we found that the average test-retest reliability for the rating scales was .65 while the average test-retest reliability for the ranked values was .43.
Bibliographic typeConferences, workshops, tutorials, presentations
Year of publication2004
Web Survey Bibliography - Thomas, R. K. (70)
- Alone in a Group: Comparison of Effects of a Group-Administered Paper-Pencil Survey Versus an Individually...; 2013; Higgins, W. B., Barlas, F. M., Pflieger, J., Thomas, R. K., Jeffery, D., Mattiko, M.
- Watch Your Language!: The Impact of the Survey Language on Bilingual Hispanics’ Response Process...; 2013; Ay, M., Gross, W., Cobb, C. L., Thomas, R. K.
- Changing of the Guard: Effects of Different Self-Administered Survey Modes on Sensitive Questions; 2013; Barlas, F. M., Higgins, W. B., Pflieger, J., Thomas, R. K., Jeffery, D., Mattiko, M.
- Response Format Effects in the Measurement of Employment; 2013; Rodkin, S., Thomas, R. K., Subias, S., Chu, C.
- Impact of Filter Questions on Estimates of Media Consumption; 2013; Cobb, C. L., Godinez, D., Thomas, R. K., Baim, J.
- Effects of Response Format on Measurement of Readership; 2013; Thomas, R. K., Cobb, C. L., Baim, J.
- Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys; 2012; Barlas, F. M., Thomas, R. K.
- A Shot in the Dark: Measurement Influence on Likelihood to Vaccination; 2012; Higgins, W. B., Thomas, R. K.
- Response Anchoring and Polarity Effects on Endorsement and Response Patterns; 2012; Higgins, W. B., Thomas, R. K.
- Evaluating the Impact of Emails and Landing Page on Web Survey Access; 2012; Falcone, A. E., Thomas, R. K., Mack, A. R.
- I Got a Feeling: Comparison of Feeling Thermometers with Verbally Labeled Scales in Attitude Measurement...; 2012; Thomas, R. K., Bremer, J.
- The River Flows: Comparison of Experimental Effect Replicability with Different Sample Sources; 2012; Thomas, R. K.
- How Likely?: Comparisons of Behavioral Intention Measurement Validity; 2012; Bremer, J., Thomas, R. K.
- Effects of response format on requalification for recontact studies; 2011; Thomas, R. K.
- Slider Scales Causing Serious Problems With Less Educated Respondents; 2011; Funke, F., Reips, U. -D., Thomas, R. K.
- Cross-country Comparisons: Effects of Scale Type and Response Style Differences; 2011; Thomas, R. K.
- A Comparison of Branching Response Formats with Single Response Formats; 2011; Thomas, R. K.
- Cleaving the Past Behind: A Comparison of Response Formats in the Measurement of Ethnic and Racial Origins...; 2011; Barlas, F. M., Thomas, R. K., Higgins, W. B.
- An Injured Party?: A Comparison of Political Party Response Formats in Party Identification.; 2011; Schwarz, S., Barlas, F. M., Thomas, R. K., Corso, R. A., Szoc, R.
- Sliders for the Smart: Type of Rating Scale on the Web Interacts With Educational Level; 2011; Funke, F., Reips, U. -D., Thomas, R. K.
- Function follows form: Effects of response format on self-reported individuals and household disability...; 2010; Falcone, A. E., Thomas, R. K.
- Response format effects on measurement of employment; 2009; Thomas, R. K., Dillman, D. A., Smyth, J. D.
- Twisting rating scales in Web surveys: Slider scales versus categorical scales of horizontal versus...; 2009; Funke, F. Reips, U. -D. Thomas, R. K.
- Increasing Confidence in Survey Estimates with Visual Analogue Scales; 2009; Funke, F., Reips, U. -D., Thomas, R. K.
- Response Formats in Cross-cultural Comparisons in Web-based Surveys; 2009; Thomas, R. K.l, Terhanian, G., Funke, F.
- Optimal Design of Branching Questions to Measure Bipolar Constructs; 2009; Malhotra, N., Krosnick, J. A., Thomas, R. K.
- Comparing Adolescent Response Bias Between Internet and Telephone Surveys ; 2009; Klein, J. D., Graff Havens, C., Thomas, R. K.
- Parallel Phone and Web-based Interviews: Effects of Sample and Weighting on Comparability and Validity...; 2008; Thomas, R. K., Krane, D., Taylor, H., Terhanian, G.
- Response Non-Differentiation and Response Styles in Web-Based Studies: Causes and Consequences ; 2008; Frisina, L. T., Thomas, R. K.
- Visual Analogue Scales in Cross Cultural Web Surveys ; 2008; Funke, F., Reips, U. -D., Thomas, R. K.
- Truth in measurement: Comparing Web Based interviewing Techniques; 2007; Couper, M. P., Terhanian, G., Bremer, J., Thomas, R. K.
- A Comparison of Visual Analog and Graphic Rating Scales ; 2007; Thomas, R. K., Couper, M. P.
- The Best of Intentions: Response Format Effects on Measures of Behavioral lntention ; 2007; Thomas, R. K., Klein, J. D., Behnke, C. S., Terhanian, G.
- Scaling Social Desirability: Establishing its Influence Across Modes; 2007; Krane, D., Thomas, R. K., Taylor, H.
- On the Importance of Form: Effects of Response Format on Measures off importance; 2007; Thomas, R. K., Behnke, C. S., Allenza, J., Klein, J. D.
- Improving importance Assessment: Experimental Comparisons between Variations of Ranking and Rating Tasks...; 2007; Thomas, R. K., Allenza, J., Behnke, C. S.
- Behavioral self-report measures. International extensions; 2006; Thomas, R. K., Klein, J. D.
- Merely Incidental?: Effects of Response Format on Self-reported Behavior; 2006; Thomas, R. K., Klein, J. D.
- Response Order Effects in International Online Surveys; 2006; Thomas, R. K., Greenfield, S., Bremer, J.
- Attitude Measurement in Phone and Online Surveys: Can Different Modes and Samples Yield Similar Results...; 2006; Thomas, R. K., Krane, D., Taylor, H., Terhanian, G.
- Response format effects in self-report of political and non-political contributions; 2005; Thomas, R. K. et al.
- Rating versus comparative trade-off measures. Trending changes in political issues across time and predictive...; 2005; Thomas, R. K. et al.
- A Comparison of an Online Card Sorting Task to a Rating Task; 2005; Thomas, R. K., Bayer, L. R., Johnson, A. M., Behnke, C. S.
- A Comparison of Presidential Candidate Vote Intention Measures in U.S. Elections; 2005; Thomas, R. K., Krane, D., Sanders, M. G., Behnke, C. S.
- To Vote or Not to Vote?: A Comparison of Vote Intention Measures; 2005; Thomas, R. K., Sanders, M. G., Smith, R., Behnke, C. S.
- Response Order Effects in Online Surveys; 2005; Thomas, R. K., Behnke, C. S., Johnson, A. M.
- How Does Social Desirability Affect Responses?: Differences in Telephone and Online Surveys; 2005; Taylor, H., Krane, D., Thomas, R. K.
- On the primacy of affect in attitude-behavior research; 2004; Thomas, R. K., Schofield, C. M.
- Measuring television viewership through a multi-method approach; 2004; Terhanian, G., Bremer, J., Delaney, T. F., Thomas, R. K.
- Behavioral Intention Measurement: International Findings; 2004; Thomas, R. K., Terhanian, G., Bayer, L. R.