Relevance & Research Question: For several years, the Federal Statistical Office (FSO) has been working on the systematic implementation of questionnaire testing. A pretest laboratory was established in 2007 and complemented by an eye tracker in 2009. Questionnaires of online surveys are now increasingly evaluated by qualitative testing methods and redesigned to reduce the burden for respondents and to increase data quality of official statistics.
Methods & Data: Pretesting online questionnaires shall improve their usability, functionality and comprehensibility. At the FSO, a three step approach is applied: Firstly, we observe eye movements and facial expressions (in real-time), while respondents deal with the questionnaire. Secondly, we conduct cognitive interviews afterwards in order to discover the reasons why respondents proceeded the way they did. Thirdly, we evaluate the process of self-completing by eye tracking data (e. g. ‘Areas of Interest’) and the sequence of mouse clicks.
Results: Each source of information has its strengths and weaknesses: Generally, it is challenging to analyze eye tracking data. It is for example difficult to assess whether a longer fixation duration indicates problems or simply a higher interest in a question. Consequently, the interpretation might be misleading without profound background knowledge. From a different angle, results derived from cognitive interviews are of minor value if the answers of respondents seem to be determined by effects like acquiescence, social desirability or limited capacity for remembering and verbalizing cognitive processes. By linking our sources of information (“triangulation”) we are able to provide more valid pretesting results and recommendations for improving online questionnaires.
Added Value: When online questionnaires are tested at the FSO, cognitive interviews are conducted after eye tracking itself. Combining both methods has given us insights into users’ behaviour when reading off screen and their expectations concerning navigation. The analyses illustrate whether respondents perceive links to detailed explanations, skip instructions or entire lists of response options. General advice is provided on wording and design principles for improving online questionnaires. Our findings lead us directly to Steve Krug’s (2006) saying: “Don’t make me think!”