Web Survey Bibliography
Relevance & Research Question: Recently, there has been a surge in the use of Amazon's Mechanical Turk as a way of recruiting participants for online research. Mechanical Turk (MT) is a mini job market, where short and easy jobs are posted by “employers” and completed by “workers” for relatively little pay. Workers are primarily motivated by payment. In economics, a field in which many online studies have recently used MT, this may be a valid recruitment method. However, from a psychology and social science perspective, it seems questionable to employ "workers" as participants. The main motivation of research participants in these fields is usually to contribute to science, help researchers, and to learn about one's self. Thus, we expected that participants recruited from MT would provide lower quality of data than participants recruited via other sources.
Methods & Data: For a large-scale test development project with 160 items, the multiple site entry technique was used to recruit samples from different sites and in different modes. Among the recruitment sites and modes were mailing lists, web study lists, blog announcements, Facebook groups, and MT. Overall, there were 1583 respondents in the study, among them 1190 recruited via MT.
Results: Participants recruited via MT had faster response times (mean difference: 14859ms/page; median difference: 3348ms/page) and answered more to the middle of response scales than participants recruited via other sources. In fact, out of the 64 items with different means, MTurkers scored more in the middle of the scale in 50 items.
Added Value: Several indicators support the hypothesis that participants recruited via MT provide lower quality data than participants from traditional sources for online research. We speculate that the root of these findings is that participants sign up as "worker." Workers respond to be paid and research participants respond to help with research. A second reason why MT workers provide lower quality data may be tied to the forums they have established where jobs are discussed, including online studies. It may well be that rumors and experiences spread in these forums lead to a decrease in data quality.
Conference Homepage (abstract)
Web Survey Bibliography - Other (471)
- Improved Dialogue Features in Web Surveys?; 2013; Lund, K.
- Research Intentions are Nothing without Technology: Mixed-Method Web Surveys and the Coberen Wall of...; 2013; Ganassali, S., Rodriguez-Santos, C.
- Reducing Response Burden for Enterprises Combining Methods for Data Collection on the Internet; 2013; Vik, T.
- Measuring Wages Worldwide: Exploring the Potentials and Constraints of Volunteer Web Surveys; 2013; Steinmetz, S., Raess, D., Tijdens, K., de Pedraza, P.
- Data Quality in PC and Mobile Web Surveys; 2013; Mavletova, A. M.
- PDAs in socio-economic surveys: instrument bias, surveyor bias or both?; 2013; Escobal, J., Benites, S.
- Virtual research assistants: Replacing human interviewers by automated avatars in virtual worlds; 2013; Hasler, B. S., Tuchman, P., Friedman, D.
- Internet-Based Recruitment to a Depression Prevention Intervention: Lessons From the Mood Memos Study...; 2013; Morgan, A. J., Jorm, A. F., Mackinnon, A. J.
- Challenges for Researchers Investigating Contraceptive Use and Pregnancy Intentions of Young Women Living...; 2013; Herbert, D. L., Loxton, D., Bateson, D., Weisberg, E., Lucke, J. C.
- Using a web-based survey tool to undertake a Delphi study: Application for nurse education research; 2013; Gill, F. J., Leslie, G. D., Grech, C., Latour, J. M.
- Comparison of web-based versus paper-and-pencil administration of a humor survey; 2013; Wang, C.-C., Cheng, C.-L.;, Liu, K.-S., Cheng, Y.-Y.
- Large-Scale Analysis and Testing; 2013; Cao, M., Zhang, Q.
- Askito: An open source Web questionnaire tool; 2013; Reips, U. -D., Heilmann, T.
- Measurement invariance and quality of composite scores in a face-to-face and a web survey; 2013; Revilla, M.
- 'Ready to complete the survey on Facebook': Web 2.0 as a research tool in business studies; 2013; Gregori, A., Baltar, F.
- Surveying “difficult-to-sample” backpackers through Facebook? Employing a mixed-mode dual...; 2013; Morris Paris, C.
- A comparison of data quality and practicality of online versus postal questionnaires in a sample of...; 2013; King, M. T., Butow, P., Olver, I., Smith, A. B.
- ISO 20252. Market, opinion and social research-Vocabulary and service requirements, 2nd Edition; 2012
- Internet use in households and by individual in 2012. Eurostat Statistics in Focus 50/2012; 2012; Seybert, H.
- Eurobarometer Special surveys: Special Eurobarometer 381; 2012
- The Impact of Academic Sponsorship on Online Survey Dropout Rates; 2012; Allen, P. J., Roberts, L. D.
- Social media as a data collection tool: the impact of Facebook in behavioural research; 2012; Zoppos, E.
- An Empirical Investigation of the Role of the Email Contact in Web Survey Response Rates; 2012; Hsu, H.-Y., Lai, Y.-H., Chin, H.-Y.
- Mode and non-response effects and their treatment; 2012; Chrysanthopoulos, S., Georgostathi, A.
- What can be said about quality in the Central Population Register based on a self-completion survey...; 2012; Falnes-Dalheim, E., Pedersen, H. E.
- Boosting Web pick-up Rates by referring to Compliance Principles ; 2012; Falnes-Dalheim, E., Haraldsen, G., Sundvoll, A.
- Subjective Well-being Of Spanish Workers: Continuous Voluntary Web Survey Examination; 2012; de Pedraza, P., Guzi, M.
- Facing The Future Webcams as a survey tool in China; 2012; Gordon, A., Llewellyn, T., Gu, E.
- Research in the Mobile Mindset: Exploring the unexplored in the mobile research space; 2012; Willems, A., Veris, E., Verhaeghe, A.
- Reality check in the digital age: The relationship between what we ask and what people actually do; 2012; Hofmeyr, J., Louw, A.
- WEBDATANET: web-based data-collection methodological challenges, solutions and implementations. Action...; 2012; de Pedraza, P.
- WebSM Study: Survey software features overview ; 2012; Vehovar, V.; Cehovin, G.; Kavcic, L.; Lenar, J.
- Reaching and Hearing the Invisible: Organizational Research on Invisible Stigmatized Groups via Web...; 2012; Trau, R. N. C., Haertel, C. E. J., Haertel, G. F.
- Response rates in three opinion surveys performed through online questionnaires in the health setting...; 2012; Aerny Perreten, N., Domínguez-Berjón, M. F., Astray Mochales, J., Esteban-Vasallo, M. D., Blanco Ancos...
- Use of Web 2.0 to Recruit Australian Gay Men to an Online HIV/AIDS Survey; 2012; Theriault, N., Bi, P., Hiller, J. E., Nor, M.
- Evaluation of an online (opt-in) panel for public participation geographic information systems surveys...; 2012; Brown, G., Weber, D., Zanon, D., de Bie, K.
- Studying Migrants with the Help of the Internet: Methods from Psychology; 2012; Reips, U. -D., Buffardi, L.
- Measures of Data Quality Across the RDD Frames; 2012; Lavrakas, P. J.
- Collecting data electronically from enterprises – searching for the right approach; 2012; Keating, J., Portillo, S.
- Psychometric properties of an internet administered version of the Marlowe-Crowne Social Desirability...; 2012; Vesteinsdottir, V., Reips, U. -D., Joinson, A. N., Porsdottir, F.
- Analysis of coverage bias for the implementation of web surveys in Spain; 2012; de Pedraza, P., Serrano, F.
- Presidential Elections in Iceland 2012 – Did online panel surveys give false hope to new candidates...; 2012; Jonsdottir, G. A., Dofradottir, A. G., Bjornsdottir, A. E.
- Website exit surveys. What can we measure with them?; 2012; Andreadis, I.
- Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and...; 2012; Sinclair, M., O'Toole, J., Malawaraarachchi, M., Leder, K.
- Online Questionnaire Data Analysis System (OQDAS); 2012; Ali, A. Q.
- Analyzing Functionalities for Online Questionnaire System (OQS); 2012; Atown, H. Y.
- Mixed Mode Survey Design: Mode Effect Problem; 2012; Sztabinski, F., Zmijewska-Jedrzejczyk, T.
- Sample Survey on Sensitive Topics: Investigating Respondents’ Understanding and Trust in Alternative...; 2012; De Schrijver, A.
- Bringing data from an online survey or spreadsheet into SPSS; 2012; Raftery, D.
- A Comparison of the Quality of Questions in a Face-to-face and a Web Survey; 2012; Revilla, M., Saris, W. E.