Web Survey Bibliography
Despite a hundred years of questionnaire testing, no consensus has been reached on the optimal number of response alternatives in rating scales. Differences in prior research may have been due to the use of various psychometric models (classical test theory, item factor analysis, and item response theory) and different performance criteria (reliability, convergent/discriminant validity, and internal structure of the questionnaire). Furthermore, previous empirical studies on this issue have tackled the experimental design from a between-subjects perspective, thus ignoring intra-individual effects. In contrast with this approach, we propose a within-subjects experimental design and a comprehensive statistical methodology using structural equation models for studying all of these aspects simultaneously, therefore increasing statistical power. To illustrate the method, two personality questionnaires were examined using a repeated measures design. Results indicated that as the number of response alternatives increased, (1) internal consistency increased, (2) there was no effect on convergent validity, and (3) goodness of fit worsened. Finally, the article assesses the practical consequences of this research for the design of future personality questionnaires.
Journal Homepage (abstract)
Web Survey Bibliography - Behavior Research Methods (25)
- A standard for test reliability in group research; 2013; Ellis, J. L.
- The viability of crowdsourcing for survey research; 2011; Behrend, T. et al.
- Why do you play? The development of the motives for online gaming questionnaire (MOGQ); 2011; Demetrovics, Z., Urbán, R., Nagygyörgy, K., Farkas, J., Zilahy, D., Mervó, B., Reindl, A., Agoston,...
- Twice random, once mixed: Applying mixed models to simultaneously analyze random effects of language...; 2011; Janssen, D. P.
- Seriousness Checks are Useful to Improve Data Validity in Online Research; 2010; Diedenhofen, D., Aust, F., Ullrich, S., Musch, J.
- A validation of well-being and happiness surveys for administration via the Internet ; 2010; Howell, R. T., Rodzon, K. S., Kurai, M., Sanchez, A. H.
- A short self-report measure of problems with executive function suitable for administration via the...; 2010; Buchanan, T., Heffernan, T. M., Parrott, A. C., Ling, J., Rodgers, J., Scholey, A. B.
- A comparison of stimulus ratings made online and in person: Gender and method effects; 2010; Barenboym, D. A., Wurm, L. H., Cano, A.
- Differences in responses to Web and paper surveys among school professionals; 2010; Yetter, G., Capaccioli, K.
- Making small effects observable: Reducing error by using visual analogue scales; 2009; Funke, F., Reips, U. -D.
- Comparison of ability tests administered online and in the laboratory; 2009; Marten Ihme, J., Lemke, F., Lieder, K., Martin, F., Müller, J. C., Schmidt, S.
- Building and managing an online panel with phpPanelAdmin; 2009; Goeritz, A.
- A decade of Internet-based data collection: Time is ripe for combining e-learning with i-science; 2009; Reips, U. -D.
- The effect of varying the number of response alternatives in rating scales: Experimental evidence from...; 2009; Maydeu-Olivares, A., Kramp, U., García-Forero, C., Gallardo-Pujol, D., Coffman, D.
- Timing accuracy of Web experiments: A case study using the WebExp software package; 2009; Keller, F., Gunasekharan, S., Mayo, N., Corley, M.
- Using the World-Wide Web to obtain large-scale word norms: 190,212 ratings on a set of 2,654 German...; 2009; Lahl, O., Goeritz, A., Pietrowsky, R., Rosenberg, J.
- Practical advice for conducting ethical online experiments and questionnaires for United States psychologists...; 2008; Barchard K. A., Williams, J.
- Individual payments as a longer-term incentive in online panels ; 2008; Goeritz, A., Wolff, H.-G., Goldstein, D. G.
- Interval-level measurement with visual analogue scales in Internet-based research: VAS generator; 2008; Reips, U. -D., Funke, F.,
- Computer-assisted self-interviews: A cost effectiveness analysis; 2008; L. Vanable, P. A., Eriksen, M. D., L.Brown, J. L.
- Dynamic online surveys and experiments with the free open-source software dynQuest ; 2007; Rademacher, J. D.M, Lippke, S.
- Pocket-sized psychology studies: Exploring daily diary software for Palm Pilots; 2006; Le, B., Choi, D.
- Using personal digital assistants (PDAs) for the collection of safety belt use data in the field; 2006; Vivoda, J. M., Eby, D. W.
- The Web experiment list: A Web service for the recruitment of participants and archiving of Internet...; 2005; Reips, U. -D., Lengler, R.
- Generic HTML Form Processor: A versatile PHP script to save Web-collected data into a MySQL database; 2005; Goeritz, A., Birnbaum, M. H.