Web Survey Bibliography
Relevance & Research Question: Many studies found that the wording of a survey question can influence the answers that respondents provide. In particular, it has been shown that vague and ambiguous terms are often interpreted idiosyncratically by respondents, and thus can introduce a systematic bias into the survey data. In addition to ambiguity, the cognitive effort required to understand survey questions may affect data quality in a similar way. Earlier research identified several problematic text features (such as low-frequency words, left-embedded syntactic structures, low syntactic redundancy) that reduce question clarity and make survey questions difficult to comprehend (e.g. Lenzner, Kaczmirek, & Lenzner, 2010). This paper extends the earlier findings and examines whether the effort required to comprehend survey questions affects data quality.
Methods & Data: An experiment was carried out in which respondents were asked to complete two Web surveys (N1=825, N2=515) at a two-week interval. Approximately half of the respondents answered questionnaires that included unclear and less comprehensible questions, the other half received control questions that were easier to comprehend. Indicators of data quality were drop-out rates, number of non-substantive responses (“Don’t know’s”), number of neutral (midpoint) responses, and over-time consistency of responses across the two surveys. In addition, respondents’ verbal intelligence and motivation were assessed to examine whether question clarity effects were moderated by these two respondent characteristics.
Results: As expected, respondents receiving unclear questions provided lower-quality responses than respondents answering more comprehensible questions. Moreover, some of these effects were more pronounced among respondents with limited verbal skills and among respondents with low motivation to answer surveys.
Added value: These findings indicate that survey results can be systematically biased if questions are difficult to understand and exceed the processing effort that respondents are willing or able to invest. Making it easy for respondents to retrieve the meaning of a survey question seems to be an important requirement for obtaining high-quality answers.
Conference Homepage (abstract) / (presentation)
Web survey bibliography - 2011 (358)
- Surveying the General Public over the Internet Using Address-Based Sampling and Mail Contact Procedures...; 2011; Messer, B. L., Dillman, D. A.
- Mobile phones as an extension of the participant observer's self: Reflections on the emergent role...; 2011; Hein, W., O'Donohoe, S., Ryan, A.
- Mixed methods designs in marketing research; 2011; Harrison, R. L., Reilly, T. M.
- Introduction to Usability Testing for Survey Research; 2011; Geisen, E., Jarrett, C.
- Utilizing Web Technology in Business Data Collection: Some Norwegian, Dutch and Danish Experiences; 2011; Haraldsen, G., Snijkers, G., Roos, M., Sundvoll, A., Vik, T., Stax, H.-P.
- E-Census 2011 Portugal: implementation and results of the Pilot Survey; 2011; Vicente, P., Rosa, A., Reis, E.
- Facebook sampling methods: some methodological proposals; 2011; Macrì, E., Tessitore, C.
- Reflections on web based data collection in a mixed mode design: the case of the EU Labour Force Survey...; 2011; Kloek, W., van der Valk, J.
- Standardising the web data collection channel at the Basque Statistics Office (EUSTAT); 2011; Prado, C., Guinea , C.
- An Experimental Investigation of Mode Effects in the Hungarian Census Test 2009; 2011; Vereczkei, Z.
- Collaborative systems for enhancing the analysis of social surveys: the Grid Enabled Specialist Data...; 2011; Lambert, P., Warner, G., Doherty, T., McCafferty, S., Watt, J., Comerford, M., Gayle, V., Tan, L., Blum...
- ILS Online Survey; 2011; Weber, C.
- Development of a Web-Based Survey for Monitoring Daily Health and its Application in an Epidemiological...; 2011; Sugiura, H., Ohkusa, Y., Akahane, M., Sano, T., Okabe, N., Imamura, T.
- Sampling v. Scale: An investigation the tension between convenience sampling, response rates, probability...; 2011; Garland, P.
- Effectiveness and consequences of various recruitment methods in psychological research: case study; 2011; Poltorak, M.
- A new approach to the analysis of survey drop-out. Results from Follow-up Surveys in the German Longitudinal...; 2011; Rossmann, J., Blumenstiel, J. E., Steinbrecher, M.
- Tracking the decision-making process – Findings from an Online Rolling Cross-Section Panel Study...; 2011; Faas, T.
- Should we use the progress bar in online surveys? A meta-analysis of experiments manipulating progress...; 2011; Callegaro, M., Yang, Y., Villar, A.
- From "Web Questions" to "Propensity Score Weighting": An Evaluation of Topics and...; 2011; Welker, M., Taddicken, M.
- Rich Profiles – Or: What's the problem with self-disclosure data?; 2011; Tress, F.
- Who are leaving our panel: panel attrition and personality traits; 2011; Marchand, M.
- Mobile Research Apps – Adding New Capabilities to Market Research; 2011; Rieber, D.
- The influence of personality traits and motives for joining on participation behavior in online panels...; 2011; Keusch, F.
- Asking sensitive questions in a recruitment interview for an online panel: the income question; 2011; Schaurer, I., Struminskaya, B., Kaczmirek, L., Bandilla, W.
- Speeders in Online Value Research: Cross-checking results of fast and slow respondents in two separate...; 2011; Beckers, T., Siegers, P., Kuntz, A.
- Effects of survey question clarity on data quality; 2011; Lenzner, T.
- Respondent Characteristics as Explanations for Uninformative Survey Response: Sources of Nondifferentiation...; 2011; Van Meurs, L., Klausch, L. T., Schoenbach, K.
- Snap judgement polling; 2011; Anderson, K., Wright, M., Wheeler, M.
- Individual differences in motivation to participate in online panels; 2011; Bruggen, E., Wetzels, M., de Ruyter, K., Schillewaert, N.
- Data Use: A systematic method for checking online questionnaires; 2011; Arbittier, J.
- Understanding the pros and cons of mixed-mode research; 2011; Mora, M.
- Visiting item non-responses in internet survey data collection; 2011; Albaum, G., Roster, C. A., Smith, S. M., Wiley, J. B.
- Why Web-assisted TDIs are a cost-effective qualitative methodology ; 2011; Donnelly, T.
- Capturing affective experiences using the SMS Experience Sampling (SMS-ES) method.; 2011; Andrews, L., Russell-Bennett, R., Drennan, J.
- Successful Prompting Methods on a Web-Based Survey; 2011; Venkataraman, L.
- Multi-Mode Survey Administration; 2011; Holder, T.
- Do’s and Don’ts of Developing Mixed Mode Surveys; 2011; Sanders, Ti.
- Mobile Survey Development Toolkit/Survey Framework; 2011; Rauch, M.
- Web based CATI on Amazon Elastic Compute Cloud and VirtualBox using queXS; 2011; Zammit, A.
- Survey Suite: Our "LOGIN & GO" Solution to Survey Research Needs; 2011; Lowden, M.
- A Dinosaur That Just Won't Die: A Return to Paper Surveys; 2011; Crandall, S., Crisafulli, T.
- Responses to Mail-Internet Mixed Mode Surveys: When Can we do Away with Paper Questionnaires?; 2011; Krebill-Prather, R.
- Web/Cloud Based CATI Using queXS; 2011; Zammit, A.
- When Referring to Mode, Is Expressed Preference the Same as Reality?; 2011; Denk, K.
- Developing Paradata Tools to Maximize Call Center Conversion Rates; 2011; Heinrich, T., Pittman, J., Abu, K.
- Incentives, Research-based Best Practices; 2011; Dykema, J.
- "But This is My Cell Phone!": A Qualitative Look at Practical Techniques for Gaining the...; 2011; George, J., Balok, T., Frasier, A. M.
- Developing and Implementing Adaptive Total Design (ATD); 2011; Carley-Baxter, L. R., Mitchell, S., Peytchev, A., Day, O.
- Three Era's of Survey Research; 2011; Groves, R. M.
- Creating Effective Designs for Mixed-Mode Surveys; 2011; Dillman, D. A.