Web Survey Bibliography
(a) Relevance & Research Question: Rating scales can considerably affect data quality regarding mean ratings, distribution of answers, response time, or item nonresponse (e.g., Couper, Conrad, & Tourangeau, 2007; Healey, 2007; Heerwegh & Loosveldt, 2002; Krosnick, 1999; Krosnick & Fabrigar, 1997). However, because implementation is easy, designers of Web surveys are tempted to use special rating scales without knowing much about their impact on data quality. This presentation focuses on how slider scales may harm survey data. Nevertheless, sometimes changes in rating scales are inevitable, especially when scrolling on Internet devices with an upright display (e.g., smart phones) should be avoided (for problems see Couper, Tourangeau, Conrad, & Crawford, 2004).
(b) Methods & Data: In a 2 x 2 Web experiment, type of rating scale (5-point Java-based slider versus 5-point HTML radio button scale) was manipulated as well as the spatial orientation on the screen. On a single Web page, respondents ( N = 779) had to evaluate two product concepts, counterbalanced for order. For analysis, respondents’ reported education was recoded in two groups, below college degree (e.g., B.A. or B.S.) and at least college degree.
(c) Results: Overall, break-off was significantly higher with slider scales in comparison to radio button scales, chi2(1, N = 779) = 12.81, p < .001, odds ratio = 6.92. Whereas respondents in the group with low education had problems with slider scales, chi2(1, N = 451) = 5.89, p = .018, odds ratio = 5.45, no difference in break-off was observed in the group of respondents with a high formal education, chi2(1, N = 321) = 1.66, p = 1.000. Additionally, task duration was considerably higher with slider scales, F (1, 703) = 638,23, p < .001, eta2 = .48. Furthermore, fewer respondents chose the middle category with slider scales. Spatial orientation of the rating scale had no significant influence on break-off or distribution of values.
(d) Added Value: The interaction between rating scale and educational level is a serious argument against the use of Java-based slider scales in general. Overall, it seems that horizontal and vertical layout can be substituted mutually.
Conference Homepage (abstract)
Web Survey Bibliography - 2011 (567)
- Developing Electronic Questionnaire Guidelines: Issues and Challenges in a Changing Environment; 2011; Cote, A.-M., Kelly, P., Lawrence, D.
- Triton: a general tool for data collection and micro editing; 2011; Erikson, J.
- A Generalized System for Aided Development and Monitoring of Web Surveys; 2011; Torelli, R.
- Ethical issues in Internet research; 2011; Hoerger, M., Currell, C.
- Using survey data collection as a tool for improving the survey process; 2011; Biffignandi, S., Perani, G., Laureti, A.
- Essential methods for design based sample surveys; 2011; Pfeffermann, D., Rao, C. R.
- ESOMAR AND CASRO submission to the W3C tracking protection working group - Market research techniques...; 2011
- A picnic in the field; negotiating the presentation of the self in researcher/respondent relationships...; 2011; Parsons, J.
- The benefits and constraints of e-mail interviews and discussions as methods of accessing valid data; 2011; Roberts, A.
- Is There a Quick Fix for Open-ended Questions? A Comparison of Qualitative Analysis Techniques; 2011; Tesfaye, C.
- The Impact of Open-Ended Questions: A Multivariate Study of Respondent Engagement; 2011; Gittelman, S. H.
- “You are Invited to Participate”: Challenges of Applying Mixed Survey Methods to Assess...; 2011; Chew, F.
- Literacy and Data Quality in Self-Administered Surveys; 2011; Smyth, J. D., Olson, K.
- Catch Them When You Can: Speeders and Their Role in Online Data Quality; 2011; Gutierrez, C., Wells, T., Rao, K., Kurzynski, D.
- Effects of Post-Incentives on Response Rates, Costs, and Response Quality in a Web Survey of College...; 2011; Stevenson, J., Dykema, J., Cyffka, K., Klein, L., Goldrick-Rab, S.
- Observed Differences in the Placement and Wording of Neutral Response Options in Web Surveys: An Experiment...; 2011; Walton, L., Cobb, C. L., DiSogra, C.
- Effects of response format on requalification for recontact studies; 2011; Thomas, R. K.
- A meta-analysis of experiments manipulating progress indicators in Web surveys; 2011; Callegaro, M., Villar, A., Yang, Y.
- Does mentioning "some people" and "other people" in a survey question increase the...; 2011; Yeager, D. S., Krosnick, J. A.
- Do not track gathers momentum; 2011; Stark, D.
- “Don’t know” the difference - An experimental comparison between Web and CATI; 2011; Schielicke, A.-M., Degen, M.
- Display resolution; 2011
- A Survey Stopping Rule Based on Weighting for Unit Nonresponse; 2011; Lewis, T.
- Classic Inspirations for Social Research Methodology in the time of Online Access Panels ; 2011; Jeřábek, H.
- Five Things You Should Know About Mobile Data Collection; 2011; Pingitore, G.
- Mixed Methods - Analyzing Open-Ended Comments in a Quantitative Employee Survey; 2011; Lawton, L., Broege, N.
- Changing Survey Methods (Discussion); 2011; Lavrakas, P. J.
- Code of standards and ethics for survey research; 2011
- Causes of survey incompletes: Why panelists say they abandon surveys; 2011; Henning, J.
- Canadian online panels: Similar or different?; 2011; Chan, P., Ambrose, D.
- Blend, balance, and stabilize respondent sources; 2011; Eggers, M., Drake, E.
- Beyond data stability: Rising above quality concerns; 2011
- Background - QSOAP; 2011
- Audience evolution: New technologies and the transformation of media audiences; 2011; Napoli, P. M.
- Assessing personality traits through response latencies using item response theory; 2011; Ranger, J., Ortner, T. M.
- American public opinion: Its origins, content, and impact (8th Edition); 2011; Erikson, R. S., Tedin, K. L.
- Amazon's Mechanical Turk. A new source of inexpensive, yet high quality, data?; 2011; Buhrmester, M., Kwang, T., Gosling, S. D.
- A theory of public opinion. A reprint of a classic book with a new introduction by H. Lee Cheek, Jr.; 2011; Wilson, F.
- Clarifying Survey Questions; 2011; Redline, C. D.
- A new representative standard for online research: Conquering the challenge of the dirty little "...; 2011; Gittelman, S., Trimarchi, E., Fawson, B.
- 2011 Skills for Life Survey: Headline findings ; 2011
- The place for mobile research? Multi-mode studies of major cultural events; 2011; Conry, S., Atkinson, S.
- Facial imaging: The new face of online survey research; 2011; Gordon, A., McCallum, D., Sorci, M., Llewellyn, T.
- How far is too far: Traditional, flash and gamification interfaces, and implications for the future...; 2011; Puleston, J., Malinoff, B.
- The Evolution of Edits in the Canadian Census of Population Online Questionnaires; 2011; Laroche, D.
- The Main Innovations of Data Editing and Imputation for the 2010 Italian Agricultural Census ; 2011; Bianchi, G., Lipsi, R. M., Ruocco, G., Salvatore, M. A.
- A classification of question characteristics relevant to measurement (error) and consequently important...; 2011; Campanelli, P., Nicolaas, G., Jaeckle, A., Lynn, P., Hope, S., Blake, M., Gray, M.
- Hrh remuneration: Comparing wage levels, ranking And dispersion of 16 occupations In the health workforce...; 2011; Tijdens, K., de Vries, D.
- Wages worldwide results and measurement issues from the multi-country. WageIndicator web-survey ; 2011; van Klaveren, M., Tijdens, K.
- Text string matching to measure occupations in web-surveys; 2011; Tijdens, K.