Web Survey Bibliography
Relevance and research question: Nonserious answering is one of the most important threats to the validity of online research (Oppenheimer, 2009). Respondents with little motivation to participate, or respondents who are interested in a survey's content or methodology only may decide to participate without giving serious answers, thus increasing noise and reducing experimental power.
Methods and data: One approach to identifying nonserious participants is to directly ask respondents about the seriousness of their participation (Klauer, Musch & Naumer, 2000; Musch & Klauer, 2002; Reips, 2000, 2002). We hypothesized that when given an opportunity to do so, randomly answering participants might be willing to identify themselves to help researchers (Reips, 2009). To validate this approach, we questioned a sample of more than 3000 respondents in the week prior to the German 2009 federal election to the Bundestag. We asked the participants whether they were responding to the questions in earnest, expecting that the exclusion of nonserious participants would help to improve data quality.
Results: We found that restricting analyses to serious participants allowed a more valid forecast of the election result. Moreover, serious participants answered attitudinal questions in a more consistent manner than nonserious participants. For example, among serious participants, self-ratings on a left-right scale correlated more strongly with approval ratings for the two major parties (CDU/CSU and SPD), and intentions to vote corresponded better with the participant’s recollections of their voting behavior in a previous election.
Added value: Taken together, our results document the usefulness of employing seriousness checks to improve data validity. We therefore recommend to routinely employ seriousness checks in online surveys. Nonserious participants should be allowed to render their data invalid, instead of letting their data invalidate the results.
Conference Homepage (abstract)
Springer Homepage (abstract) / (full text)
Web survey bibliography - Behavior Research Methods (16)
- Use of Internet panels to conduct surveys; 2015; Kapteyn, A.; Liu, H., D.Hays, R. D.
- QRTEngine: An easy solution for running online reaction time experiments using Qualtrics; 2014; Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., van Steenbergen, H.
- Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis; 2014; Gnambs, T., Kaspar, K.
- Build your own social network laboratory with Social Lab: a tool for research in social media; 2014; Garaizar, P., Reips, U.-D.
- A standard for test reliability in group research; 2013; Ellis, J. L.
- Why do you play? The development of the motives for online gaming questionnaire (MOGQ); 2011; Demetrovics, Z., Urbán, R., Nagygyörgy, K., Farkas, J., Zilahy, D., Mervó, B., Reindl, A., Agoston,...
- Seriousness Checks are Useful to Improve Data Validity in Online Research; 2010; Diedenhofen, D., Aust, F., Ullrich, S., Musch, J.
- Differences in responses to Web and paper surveys among school professionals; 2010; Yetter, G., Capaccioli, K.
- Making small effects observable: Reducing error by using visual analogue scales; 2009; Funke, F., Reips, U.-D.
- Comparison of ability tests administered online and in the laboratory; 2009; Marten Ihme, J., Lemke, F., Lieder, K., Martin, F., C. C., Schmidt, Sa., Mueller, J. C.
- A decade of Internet-based data collection: Time is ripe for combining e-learning with i-science; 2009; Reips, U.-D.
- The effect of varying the number of response alternatives in rating scales: Experimental evidence from...; 2009; Maydeu-Olivares, A., Kramp, U., García-Forero, C., Gallardo-Pujol, D., Coffman, D.
- Timing accuracy of Web experiments: A case study using the WebExp software package; 2009; Keller, F., Gunasekharan, S., Mayo, N., Corley, M.
- Using the World-Wide Web to obtain large-scale word norms: 190,212 ratings on a set of 2,654 German...; 2009; Lahl, O., Goeritz, A., Pietrowsky, R., Rosenberg, J.
- Individual payments as a longer-term incentive in online panels ; 2008; Goeritz, A., Wolff, H.-G., Goldstein, D. G.
- The Web experiment list: A Web service for the recruitment of participants and archiving of Internet...; 2005; Reips, U. -D., Lengler, R.