Web Survey Bibliography
Relevance & Research Question: A matrix question format (grid) has may advantages in traditional paper-and-pen surveys design: more questions can be presented in a smaller space, thereby saving paper, printing and mailing costs. With the onset of computer-assisted questionnaires and Internet research, these cost saving arguments are no longer valid. Still, matrix questions remain a widely used tool also in Web surveys (e.g. Couper, 2008). There is some empirical evidence (e.g. Toepoel et al 2005; Petchev, 2006; Callegaro, 2010) that matrix questions lead to more statisficing and more dropout. However, presenting questions as single items on the screen leads to longer response times and may increase response burden. In this study we therefore evaluate an alternative to traditional matrix and scrolling formats for web surveys: the horizontal scrolling matrix (HSM).
Methods & Data: In the HSM questions are presented one-by-one on the screen, while ease of answering is improved (e.g., after an answer is selected, the next question is quickly and automatically presented, there is no need for the next or previous button, etc.), and respondents still have an overview of the number of questions in the matrix through a visual bar. In a randomized field experiment we tested whether the HSM question format does improve data quality over the traditional matrix format. We used a 2 by 3 factorial design: the experimental factors were (1) question format: traditional matrix vs HSM and (2) length of ‘matrix’: 5, 10, and 15 items per matrix.
Results: The results were positive: the HSM-format led to fewer break-offs, better data quality and less satisficing as indicated by well-known response styles, such as non-differentiation and extremeness. The HSM helped improving data quality in the longer matrix conditions, which typically suffer from worse data quality. Finally, respondents evaluated the HSM format very positive.
Added Value: Online researchers can profit from the flexibility of web surveys by moving away from grid formats, which are a heritage of paper and pencil surveys. The HSM is a useful alternative, which prevents respondent demotivation by offering superior usability.
GOR Homepage (abstract) / (full text)
Web survey bibliography - De Leeuw, E. D. (27)
- Estimating the Impact of Measurement Differences Introduced by Efforts to Reach a Balanced Response...; 2017; Kappelhof, J. W. S.; De Leeuw, E. D.
- Mixed Mode Research: Issues in Design and Analysis; 2016; Hox, J.; De Leeuw, E. D.; Klausch, L. T.
- Dropouts in Longitudinal Surveys; 2016; Lugtig, P. J.; De Leeuw, E. D.
- Internet Panels, Professional Respondents, and Data Quality; 2015; Matthijsse, S.; De Leeuw, E. D.; Hox, J.
- Mode System Effects in an Online Panel Study: Comparing a Probability-based Online Panel with two Face...; 2015; Struminskaya, B.; De Leeuw, E. D.; Kaczmirek, L.
- Pret met panels [Fun online]; 2013; Roberts, A., de Leeuw, E. D., Hox, J., Klausch, L. T., de Jongh, A.
- Leuker kunnen wij het wel maken. Online vragenlijst design: standaard matrix of scrollmatrix (We can...; 2013; Roberts, A., de Leeuw, E. D., Hox, J., Klausch, L. T., de Jongh, A.
- Internet Coverage and Coverage Bias in Europe: Developments Across Countries and Over Time; 2013; Mohorko, A., de Leeuw, E. D.,Hox, J.
- Random versus Systematic Error in a Mixed Mode Online-Telephone Survey; 2013; Hox, J., Scherpenzeel, A., Boeve, A., Boeve, A., de Leeuw, E. D.
- Mode Effects in Mixed-Mode Surveys: Prevention, Diagnostics, and Adjustment 1; 2013; de Leeuw, E. D., Dillman, D. A., Schouten, B.
- Does one really know?: Avoiding noninformative answers in a reliable way.; 2013; de Leeuw, E. D., Boevee, A., Hox, J.
- Counting and Measuring Online: The Quality of Internet Surveys; 2012; De Leeuw, E. D.
- Design of Web Questionnaires: Matrix Questions or Single Question Formats ; 2012; de Leeuw, E. D., Hox, J., Klausch, L. T., Roberts, A., de Jongh, A.
- Professional Respondents in Internet Panels: Who are They and What Do They Do to Our Data?; 2012; de Leeuw, E. D., Matthijsse, S.
- Question or Mode Effects in Mixed-Mode surveys: A Cross-cultural study in the Netherlands, Germany,...; 2012; de Leeuw, E. D., Nicolaas, G., Campanelli, P., Hox, J.
- Matrix vs. Single Question Formats in Web Surveys: Results from a large scale experiment; 2012; Klausch, L. T., de Leeuw, E. D., Hox, J., de Jongh, A., Roberts , A.
- Flexibility of Web Surveys: Probing 'do-not-know' over the Phone and on the Web; 2011; Hox, J., de Leeuw, E. D.
- Mode Effect or Question Wording? Measurement Error in Mixed Mode Surveys; 2011; de Leeuw, E. D., Hox, J., Scherpenzeel, A.
- Measurement Error in Mixed Mode Surveys: Mode or Question Format?; 2011; de Leeuw, E. D., Hox, J.
- Online Interviewing through Access Panel: Quantity and Quality Assurance; 2009; Petric, I., Appel, M., de Leeuw, E. D.
- Reducing Measurement Errors in Surveys; 2009; De Leeuw, E. D.
- Missing data; 2008; de Leeuw, E. D., Hox, J.
- The influence of advance letters on response in telephone surveys; 2007; de Leeuw, E. D., Callegaro, M., Hox, J., Korendijk, E., Lensvelt-Mulders, G. J.
- Have Telephone Surveys a Future in the 21-th century?; 2002; de Leeuw, E. D., Lepkowski, J. M., Kim, S.-W.
- Trends in household survey nonresponse: A longitudinal and international comparison; 2001; de Leeuw, E. D., de Heer, W.
- The effect of computer-assisted interviewing on data quality: A review.; 1995; de Leeuw, E. D., Hox, J., Snijkers, G.
- Data Quality in Mail, Telephone and Face to Face Surveys; 1992; De Leeuw, E. D.