The seriousness check (Reips, 2000, 2008) consists of one or two questions asked at the very beginning of an Internet-based study (“Have you already participated in this study?”, “Are you going to fill in the questionnaire seriously?”). This gives participants the chance to state their motivation in taking part. Some are, for example, researchers or students and thus only interested in the study design and technology behind the online study. These people should have the possibility to mark their data as invalid. In the present study we empirically validate the seriousness technique approach.

In a large online questionnaire study, a total of 1,046 data sets were collected. In order to get a measure of data quality, we recorded user actions (clicks, mouse movements, activation of form elements like radio buttons, text input) with time stamps by using a Java script that was implemented on each HTML page. User actions (e.g., clicking through, excessive clicking, longer inactivities, or changes on already given answers) were inspected for suspicious and highly suspicious behaviors.

A correlation analysis showed that participants who stated having already participated in the study also had more highly suspicious behaviors (r = .113, p < .001) and participants who stated not going to fill in the questionnaire seriously also showed more suspicious (r = .080, p = .011) and highly suspicious behaviors (α = 10%; r = .051, p = .099). Sorting out previous and non-serious participants reduced the amount of highly suspicious behaviors (mean drop of 11.9%).

The approach used in this study made it possible to add to our knowledge about the effectiveness of the seriousness check. It turned out that asking filter questions indeed reduces problems with data quality in online questionnaires.