Relevance & Research Question:

While tracking mouse movements is common in other areas of usability testing (e.g., web design, e-learning), applying mouse movement tracking as a tool for web questionnaire testing is relatively new and has so far been mostly limited to lab studies. In the current study, we operationalize collection of specific movements on a large scale outside the lab and we experimentally vary the type of difficulty in survey questions to see if different movements are associated with different cognitive processing.

Methods & Data:

The data for this study come from a web survey among 1,250 people who are employed, unemployed, job seekers, recipients of unemployment benefit II, and active labor market program participants. The study was conducted by the Institute for Employment Research in Nuremberg, Germany, in fall 2016. The questionnaire includes factual, opinion, and problem-solving questions with a variety of response formats, such as radio buttons and slide bars. We vary experimentally the difficulty and complexity of items between respondents to show how complexity affects behavior. We collect and log participants' mouse movements as they complete the online survey.

Results:

We find that unsorted lists of response questions are associated with more mouse movements and more vertical regressions than sorted lists. We also find that Yes/No format results in more mouse movements and more horizontal flips than check all that apply questions. We also find specific patterns of mouse movements on sensitive questions and when difficult terms are used.

Added Value:

By collecting and analyzing participants' mouse movement data during completion of the questionnaire, we show how complexity affects response behavior and the degree to which indicators of insecurity are related to the veracity of answers. Our results constitute initial steps toward a real-time analysis of the collected paradata, and provide a building-block for adaptive questionnaires that employ online detection and resolution of respondent's difficulties, leading to more accurate survey data.