Web Survey Bibliography
Relevance & Research Question: For several years, the Federal Statistical Office (FSO) has been working on the systematic implementation of questionnaire testing. A pretest laboratory was established in 2007 and complemented by an eye tracker in 2009. Questionnaires of online surveys are now increasingly evaluated by qualitative testing methods and redesigned to reduce the burden for respondents and to increase data quality of official statistics.
Methods & Data: Pretesting online questionnaires shall improve their usability, functionality and comprehensibility. At the FSO, a three step approach is applied: Firstly, we observe eye movements and facial expressions (in real-time), while respondents deal with the questionnaire. Secondly, we conduct cognitive interviews afterwards in order to discover the reasons why respondents proceeded the way they did. Thirdly, we evaluate the process of self-completing by eye tracking data (e. g. ‘Areas of Interest’) and the sequence of mouse clicks.
Results: Each source of information has its strengths and weaknesses: Generally, it is challenging to analyze eye tracking data. It is for example difficult to assess whether a longer fixation duration indicates problems or simply a higher interest in a question. Consequently, the interpretation might be misleading without profound background knowledge. From a different angle, results derived from cognitive interviews are of minor value if the answers of respondents seem to be determined by effects like acquiescence, social desirability or limited capacity for remembering and verbalizing cognitive processes. By linking our sources of information (“triangulation”) we are able to provide more valid pretesting results and recommendations for improving online questionnaires.
Added Value: When online questionnaires are tested at the FSO, cognitive interviews are conducted after eye tracking itself. Combining both methods has given us insights into users’ behaviour when reading off screen and their expectations concerning navigation. The analyses illustrate whether respondents perceive links to detailed explanations, skip instructions or entire lists of response options. General advice is provided on wording and design principles for improving online questionnaires. Our findings lead us directly to Steve Krug’s (2006) saying: “Don’t make me think!”
Conference Homepage (abstract)
Web survey bibliography - General Online Research Conference (GOR) 2010 (17)
- Testing the Applicability of Respondent Driven Sampling as an Online Research Method to Sample Hidden...; 2010; Pajak, D.
- Seriousness Checks are Useful to Improve Data Validity in Online Research; 2010; Diedenhofen, D., Aust, F., Ullrich, S., Musch, J.
- Enrichment of Qualitative Research through Online Approaches: New Insights due to Online CoCreation...; 2010; Krischke-Ramaswamy, M., Knorr, H.
- Developing and Evaluating a Student Online Panel.; 2010; Stiglbauer, B., Gamsjäger, M., Gnambs, T., Batinic, B., Altrichter, H.
- Online Access Panels: A detailed look at different Ways of Entering, their Costs and Participation Behavior...; 2010; Führer, R., Keusch, F.
- Eye Tracking and Cognitive Interviewing: Steps to improve online questionnaires; 2010; Tries, S., Sattelberger, S.
- Trial by Ordeal, a medieval approach to a modern day problem; 2010; Cape, P., Cavallaro, K.
- How new engagement techniques and question approaches are revolutionizing online research data gathering...; 2010; Puleston, J.
- Social Networking Sites: New approaches for Online-Panels?; 2010; Drosdow, M., Geißler, H.
- The Impact of Visual and Functional Design Elements in Online Survey Research; 2010; Hammen, K.
- Theoretical model of context-sensitive mobile methods; 2010; Maxl, E.
- Can a professional questionnaire layout make up for a boring topic? The mediating role of topic interest...; 2010; Keusch, F., Mayerhofer, W., Jungreithmaier, S., Weilbuchner, N., Fuehrer, R., Kling, H.
- Using Propensity Score Weighting to Reduce Bias of a Swiss Market Research Web Panel; 2010; Wiegand, G., Jella, H., Beat, H., Stefan, L.
- Potentials and Constraints of Propensity Score Weighting to Improve Web Survey Quality; 2010; Steinmetz, S., Tijdens, K.
- Selection Bias in Web Surveys and the Use of Propensity Scores in Forecasting the Result of the 2009...; 2010; Musch, J., Ullrich, S., Diedenhofen, D.
- Breakoff in Web Surveys of the German Longitudinal Election Study (GLES); 2010; Blumenstiel, J. E., Roßmann, J., Steinbrecher, M.
- The longitudinal effect of incentives on participation and data quality in online panels; 2010; Neumann, B. P., Goeritz, A.