Web Survey Bibliography
Relevance & Research Question: Many studies found that the wording of a survey question can influence the answers that respondents provide. In particular, it has been shown that vague and ambiguous terms are often interpreted idiosyncratically by respondents, and thus can introduce a systematic bias into the survey data. In addition to ambiguity, the cognitive effort required to understand survey questions may affect data quality in a similar way. Earlier research identified several problematic text features (such as low-frequency words, left-embedded syntactic structures, low syntactic redundancy) that reduce question clarity and make survey questions difficult to comprehend (e.g. Lenzner, Kaczmirek, & Lenzner, 2010). This paper extends the earlier findings and examines whether the effort required to comprehend survey questions affects data quality.
Methods & Data: An experiment was carried out in which respondents were asked to complete two Web surveys (N1=825, N2=515) at a two-week interval. Approximately half of the respondents answered questionnaires that included unclear and less comprehensible questions, the other half received control questions that were easier to comprehend. Indicators of data quality were drop-out rates, number of non-substantive responses (“Don’t know’s”), number of neutral (midpoint) responses, and over-time consistency of responses across the two surveys. In addition, respondents’ verbal intelligence and motivation were assessed to examine whether question clarity effects were moderated by these two respondent characteristics.
Results: As expected, respondents receiving unclear questions provided lower-quality responses than respondents answering more comprehensible questions. Moreover, some of these effects were more pronounced among respondents with limited verbal skills and among respondents with low motivation to answer surveys.
Added value: These findings indicate that survey results can be systematically biased if questions are difficult to understand and exceed the processing effort that respondents are willing or able to invest. Making it easy for respondents to retrieve the meaning of a survey question seems to be an important requirement for obtaining high-quality answers.
Conference Homepage (abstract) / (presentation)
Web Survey Bibliography (6797)
- Exploring patterns of academic usage: A Google Scholar based study of ESS, EVS, WVS and ISSP academic...; 2013; Malnar, B.
- Adding Observation to Copy Testing with Kids; 2013; Hurley, B., Salo, M.
- Web questionnaires in official population surveys: Do's and don'ts First experiments and impacts...; 2013; Blanke, K.
- Gamified Surveys: Journey Into Product Development; 2013; Adamou, B.
- Phish Rising: How Internet Criminals are Undermining the Viability of Online Survey Research…and...; 2013; Kunovic, K.
- The State of Technology in Market Research; 2013; Macer, T.
- Avatare in der Marktforschung: Effekte künst-licher Interviewer im Online-Interview; 2013; Luetters, H.
- Möglichkeiten zur impliziten Messung von Emotionen am Beispiel webcambasierter Gesichtsausdruckserkennung...; 2013; Wachenfeld, A., Moentmann, A., Bernet, F.
- Historic Resource Surveys and the Internet; 2013; The Center for Historic Architecture University of Houston Gerald D. Hines College of Architecture
- Mode effects in Labour Force Surveys - do they really matter?; 2013; Koerner, T.
- Effects of Sponsorship on Response: Mixed-Mode Web and Mail Surveying ; 2013; Edwards, M. L., Dillman, D. A., Smyth, J.
- Identifying and Mitigating Satisficing in Web Surveys: Some Experimental Evidence; 2013; Rossmann, J.
- Does left still feel right? The optimal position of answer boxes in Web surveys - revisited; 2013; Lenzner, T., Kaczmirek, L.,Galesic, M.
- Latent legitimacy: joint effects of religious orientation on the association between values and acceptance...; 2013; Henseler, A. K., Siegers, P., Beckers, T.
- Identifying Satisficing Respondents in Web Surveys: A Comparison of Different Response Time-Based Approaches...; 2013; Rossmann, J.
- Comparison of quality of web survey and CATI data using unobtrusive response latencies; 2013; Mayerl, J.
- A walking exercise on a tightrope: trying to balance coverage, probabilistic sampling and the costs...; 2013; Poggio, T.
- Comparability of substantive results between modes and incentive conditions in a probability-based telephone...; 2013; Pekari, N.
- Designing a Pilot to Introduce Web Collection in the UK Labour Force Survey; 2013; Merad, S.
- Data Collection Method Comparisons for the 2011 Fishing, Hunting, and Wildlife-Associated Recreation...; 2013; Herbstritt, M., Hornick, D.
- Trends in Dutch egalitarian attitudes, 1975-2010: An age-period-cohort analysis; 2013; Meuleman, B., Oorschot, W. V.
- Comparing Questionnaire Pretesting Methods for the 2015 Population and Housing Census of Korea; 2013; Park, Y.
- Evaluating the middle category in the left-right self-placement question: Category Selection Probing...; 2013; Hufken, V., Rottinghaus, B.
- Mixed Methods in Cognitive Interviewing: An analysis of the Russian version of the Schwartz Value Survey...; 2013; Efremova, M., Panyusheva, T.
- A mixed methods approach to investigate how Cognitive Interviewing findings can improve the validity...; 2013; Padilla, J. L., Miller, K., Loeb, M.
- Exploring the results of pre-testing questions developed for the European Social Survey: Comparing...; 2013; Widdop, S.
- Validity of Measuring Educational Attainment - Education and Value Orientation in the European Values...; 2013; Ortmanns, V.
- The comparability of Don't Know answers between CATI and CAWI modes; 2013; Pohjanpaa, K., Jarvensivu, M.
- Measuring the same concepts in several modes in the "BIBB/BAuA-Employee-Survey 2011/12" ; 2013; Gensicke, M., Tschersich, N., Hartmann, J.
- Error Prevention through Interviewer Emulation? Introducing questionnaire dialogues in the Norwegian...; 2013; Gravem, D. F.
- Multiple Group Confirmatory and Exploratory Factor Analysis: Exploring Measurement Invariance by Rotational...; 2013; Asparouhov, T.
- BSEM Measurement Invariance Analysis; 2013; Muthen, B. O.
- Leveraging Auxiliary Data, Differential Incentives, and Survey Mode to Target Hard-to-Reach Groups in...; 2013; Link, M. W., Thu Burks, A.
- Does It Pay Off to Include Non-Internet Households in an Internet Panel? ; 2013; Leenheer, J., Scherpenzeel, A.
- ReCal OIR: Ordinal, Interval, and Ratio Intercoder Reliability as a Web Service; 2013; Freelon, D. G.
- Classifying Mouse Movements and Providing Help in Web Surveys; 2013; Horwitz, R.
- Video Interviewing: An Exploration of the Feasibility as a Mode of Survey Application; 2013; Jeannis, M., Terry, T. L., Heman-Ackah, R., Price, M.
- Matching Data Collection Method to Purpose: In the Moment Data Collection with Mobile Devices for Occasioned...; 2013; Shea, C., Roberts, M., Johnson, E. P., Hadlock, W.
- Filling the Void: Gaining a Better Understanding of Tablet-based Surveys; 2013; Wells, T., Bailey, J., Link, M. W.
- Old versus New: The Comparative Efficiency of Mail and Internet Surveys of State Legislators.; 2013; Fisher, S. H., Herrick, R.
- Mail and Web-Based Survey Administration: A Case Study With Recreational Users of Virginia's Wildlife...; 2013; Carrozzino-Lyon, A. L., McMullin, S. L., Parkhurst, J. A.
- The 2012 Confirmit Annual Market Research Software Survey; 2013; Macer, T., Wilson, S.
- A mixed-mode survey tackling against an increasing rate of nonresponse; 2013; Jokinen, M.
- Using Mixed-Mode Contacts in Client Surveys: Getting More Bang for Your Buck; 2013; Israel, G. D.
- Using Eye Tracking Data to Understand Respondent's Processing of Rating Scales; 2013; Kunz, T., Fuchs, M.
- Advancing the field of questionnaire translation - identifying problems, discussing methods, pushing...; 2013; Behr, D., Dorer, B., Van Houten, G
- European Values Study - methodological and substantive applications; 2013; Luijkx, R., Jagodzinski, W.
- The Impact of Culture and Economy on Values and Attitudes; 2013; Duelmer, H., Voicu, M.
- Educational attainment in cross-national surveys: instrument design, data collection, harmonisation...; 2013; Schneider, S.
- Mode Effects in Mixed-Mode Surveys: Prevention, Diagnostics, and Adjustment 1; 2013; de Leeuw, E. D., Dillman, D. A., Schouten, B.