Relevance & Research Question: Recently, there has been a surge in the use of Amazon's Mechanical Turk as a way of recruiting participants for online research. Mechanical Turk (MT) is a mini job market, where short and easy jobs are posted by “employers” and completed by “workers” for relatively little pay. Workers are primarily motivated by payment. In economics, a field in which many online studies have recently used MT, this may be a valid recruitment method. However, from a psychology and social science perspective, it seems questionable to employ "workers" as participants. The main motivation of research participants in these fields is usually to contribute to science, help researchers, and to learn about one's self. Thus, we expected that participants recruited from MT would provide lower quality of data than participants recruited via other sources.
Methods & Data: For a large-scale test development project with 160 items, the multiple site entry technique was used to recruit samples from different sites and in different modes. Among the recruitment sites and modes were mailing lists, web study lists, blog announcements, Facebook groups, and MT. Overall, there were 1583 respondents in the study, among them 1190 recruited via MT.
Results: Participants recruited via MT had faster response times (mean difference: 14859ms/page; median difference: 3348ms/page) and answered more to the middle of response scales than participants recruited via other sources. In fact, out of the 64 items with different means, MTurkers scored more in the middle of the scale in 50 items.
Added Value: Several indicators support the hypothesis that participants recruited via MT provide lower quality data than participants from traditional sources for online research. We speculate that the root of these findings is that participants sign up as "worker." Workers respond to be paid and research participants respond to help with research. A second reason why MT workers provide lower quality data may be tied to the forums they have established where jobs are discussed, including online studies. It may well be that rumors and experiences spread in these forums lead to a decrease in data quality.