For several years, the Federal Statistical Office (FSO) has been working on the systematic implementation of questionnaire testing. A pretest laboratory was established in 2007 and complemented by an eye tracker in 2009. Questionnaires of online surveys are now increasingly evaluated by qualitative testing methods and redesigned to reduce the burden for respondents and to increase data quality of official statistics.

Pretesting online questionnaires shall improve their usability, functionality and comprehensibility. At the FSO, a three step approach is applied: Firstly, we observe eye movements and facial expressions (in real-time), while respondents deal with the questionnaire. Secondly, we conduct cognitive interviews (using mainly the methods of retrospective think aloud and probing) afterwards in order to discover the reasons why respondents have proceeded the way they did. Thirdly, we evaluate the process of self-completing by eye tracking data (e. g. 'Areas of Interest’) and the sequence of mouse clicks. By linking our sources of information (“triangulation”) we are able to provide more valid pretesting results and recommendations for improving online questionnaires.

Eye Tracking provides objective results on the one hand and furthermore shows off additional insights into the (often) unconscious behaviour of probands. Examples from pretests, concerning the perception and understanding of skip instructions, error messages, explanations and navigation will illustrate the benefits of eye tracking in questionnaire testing.