Amazon Mechanical Turk

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

Amazon Mechanical Turk (MTurk) is a crowdsourcing website for businesses (known as Requesters) to hire remotely located "crowdworkers" to perform discrete on-demand tasks that computers are currently unable to do. It is operated under Amazon Web Services, and is owned by Amazon.[1] Employers post jobs known as Human Intelligence Tasks (HITs), such as identifying specific content in an image or video, writing product descriptions, or answering questions, among others. Workers, colloquially known as Turkers or crowdworkers, browse among existing jobs and complete them in exchange for a rate set by the employer. To place jobs, the requesting programs use an open application programming interface (API), or the more limited MTurk Requester site.[2] As of April 2019, Requesters could register from only 49 approved countries.[3]


The service was conceived by Venky Harinarayan in a US patent disclosure in 2001.[4] Amazon coined the term artificial artificial intelligence for processes outsourcing some parts of a computer program to humans, for those tasks carried out much faster by humans than computers. It is claimed that Jeff Bezos was responsible for the concept that led to Amazon's Mechanical Turk being developed to realize this process.[5]

The name Mechanical Turk was inspired by "The Turk", an 18th-century chess-playing automaton made by Wolfgang von Kempelen that toured Europe, beating both Napoleon Bonaparte and Benjamin Franklin. It was later revealed that this "machine" was not an automaton at all, but was, in fact, a human chess master hidden in the cabinet beneath the board and controlling the movements of a humanoid dummy. Likewise, the Mechanical Turk online service uses remote human labor hidden behind a computer interface to help employers perform tasks that are not possible using a true machine.

MTurk was launched publicly on November 2, 2005. Following its launch, the Mechanical Turk user base grew quickly. In early- to mid-November 2005, there were tens of thousands of jobs, all of them uploaded to the system by Amazon itself for some of its internal tasks that required human intelligence. HIT types have expanded to include transcribing, rating, image tagging, surveys, and writing.

In March 2007, there were reportedly more than 100,000 workers in over 100 countries.[6] This increased to over 500,000 registered workers from over 190 countries in January 2011.[7] In the same year, Techlist published an interactive map pinpointing the locations of 50,000 of their MTurk workers around the world.[8] By 2018, research had demonstrated that while there were over 100,000 workers available on the platform at any time, only around 2000 were actively working.[9]

Legal status[edit]

In US labor law, according to Professor Miriam Cherry, Saint Louis University School of Law, workers on Mechanical Turk are "no different than construction workers who show up at job sites and work for a day or two on a project". Those construction workers can still file a lawsuit under the Fair Labor Standards Act for wage theft, even though they are not considered employees under their contract with Amazon.[10]


A user of Mechanical Turk can be either a "Worker" (contractor) or a "Requester" (employer). Workers have access to a dashboard that displays three sections: total earnings, HIT status and HIT totals. Workers set their own hours and are not under any obligation to accept any particular task. Amazon classifies the Workers as contractors rather than employees, and refuses to file forms or pay payroll taxes. This is aimed to evade the minimum wage, overtime, and workers compensation. Workers must report their income as self-employment income. In 2013, the average wage for the multiple microtasks assigned, if performed quickly, is about one dollar an hour, with each task averaging a few cents.[11] Workers can have a postal address anywhere in the world. Payment for completing tasks can be redeemed on via gift certificate (gift certificates are the only payment option available to international workers, apart from India) or be later transferred to a Worker's U.S. bank account.

Requesters can ask that Workers fulfill qualifications before engaging in a task, and they can set up a test in order to verify the qualification. They can also accept or reject the result sent by the Worker, which affects the Worker's reputation. As of April 2019, Requesters paid Amazon a minimum 20% commission on the price of successfully completed jobs, with increased amounts for additional services.[6] Requesters can use the Amazon Mechanical Turk API to programmatically integrate the results of that work directly into their business processes and systems. When employers set up their job, they must specify

  • how much are they paying for each HIT accomplished,
  • how many workers they want to work on each HIT,
  • maximum time a worker has to work on a single task,
  • how much time the workers have to complete the work,

as well as the specific details about the job they want to be completed.

Location of Turkers[edit]

Workers have been primarily located in the United States since the platform's inception[12] with demographics generally similar to the overall Internet population in the US.[13]

In 2010, cash payments for Indian workers were introduced, which gave new and updated results on the demographics of workers, who remained primarily within the United States.[14] The researcher behind these statistics runs a website showing worker demographics, updated hourly. In May 2015, it showed that 80% of workers were located in the United States, with the remaining 20% located elsewhere in the world, most of whom were in India.[15] As of May 2019, it showed that approximately 60% of workers were located in the United States and 40% are located elsewhere in the world; approximately 30% are in India.[16]


Human-subject research[edit]

Beginning in 2010, numerous researchers have explored the viability of Mechanical Turk to recruit subjects of social science experiments. Thousands of papers that rely on data collected from Mechanical Turk workers are published each year, including hundreds in top ranked academic journals.[17] Researchers generally found that while samples of respondents obtained through Mechanical Turk do not perfectly match all relevant characteristics of the US population, they are also not wildly misrepresentative.[18][19] A study published in 2021 found that the types of quality control approaches used by researchers (such as checking for bots, VPN users, or workers willing to submit dishonest responses) can meaningfully influence survey results, demonstrating so through impact on three common behavioral/mental healthcare screening tools.[20]

The general consensus among researchers is that the service works best for recruiting a diverse sample; it is less successful with studies that require more precisely defined populations or that require a representative sample of the population as a whole.[17] However, there are concerns that the proprietary selection algorithm may prejudice results (see: Research validity). Overall, the US MTurk population is mostly female and white, and is somewhat younger and more educated than the US population overall. Data collected on jobs conducted since 2013 show that the US population is no longer predominantly female, and that Workers are currently slightly more likely to be male.[21] The cost of MTurk was considerably lower than other means of conducting surveys, with workers willing to complete tasks for less than half the US minimum wage.[22]

Machine Learning[edit]

Supervised Machine Learning algorithms require large amounts of human-annotated data to be trained successfully. Machine learning researchers have hired Workers through Mechanical Turk to produce datasets such as SQuAD, a question answering dataset.[23]

Missing persons searches[edit]

Since 2007, the service has been used to search for prominent missing individuals. It was first suggested during the search for James Kim, but his body was found before any technical progress was made. That summer, computer scientist Jim Gray disappeared on his yacht and Amazon's Werner Vogels, a personal friend, made arrangements for DigitalGlobe, which provides satellite data for Google Maps and Google Earth, to put recent photography of the Farallon Islands on Mechanical Turk. A front-page story on Digg attracted 12,000 searchers who worked with imaging professionals on the same data. The search was unsuccessful.[24]

In September 2007, a similar arrangement was repeated in the search for aviator Steve Fossett. Satellite data was divided into 85 squared meter sections, and Mechanical Turk users were asked to flag images with "foreign objects" that might be a crash site or other evidence that should be examined more closely.[25] This search was also unsuccessful. The satellite imagery was mostly within a 50-mile radius,[26] but the crash site was eventually found by hikers about a year later, 65 miles away.[27]

Artistic works[edit]

In addition to receiving growing interest from the social sciences, MTurk has also been used as a tool for artistic creation. One of the first artists to work with Mechanical Turk was xtine burrough, with The Mechanical Olympics (2008),[28][29] Endless Om (2015) and Mediations on Digital Labor.[30][31] Another early work was artist Aaron Koblin's Ten Thousand Cents (2008).

Third-party programming[edit]

Programmers have developed various browser extensions and scripts designed to simplify the process of completing jobs. Amazon has stated that they disapprove of scripts that completely automate the process and preclude the human element. This is because of the concern that the task completion process - e.g. answering a survey - could be gamed with random responses, and the resultant collected data could be worthless.[32] Accounts using so-called automated bots have been banned. There are services that extend the capabilities to MTurk.


Amazon makes available an application programming interface (API) to give users another access point into the MTurk system. The MTurk API lets a programmer access numerous aspects of MTurk like submitting jobs, retrieving completed work, and approving or rejecting that work.[33] In 2017, Amazon launched support for AWS Software Development Kits (SDK), allowing for nine new SDKs available to MTurk Users. MTurk is accessible via API from the following languages: Python, JavaScript, Java, .NET, Go, Ruby, PHP or C++.[34] Web sites and web services can use the API to integrate MTurk work into other web applications, providing users with alternatives to the interface Amazon has built for these functions.

Worker-HIT-assignment ER_diagram

Use case examples[edit]

Processing photos / videos[edit]

Amazon Mechanical Turk provides a platform for processing images, a task well-suited to human intelligence. Requesters have created tasks asking workers to label objects found in an image, select the most relevant picture in a group of pictures, screen inappropriate content, and classify objects in satellite images. Also, crowdworkers have completed tasks of digitizing text from images such as scanned forms filled out by hand.[35]

Data cleaning / verification[edit]

Companies with large online catalogues use Mechanical Turk to identify duplicates and verify details of item entries. Some examples of fixing duplicates are identifying and removing duplicates in yellow pages directory listings and online product catalog entries. Examples of verifying details include checking restaurant details (e.g. phone number and hours) and finding contact information from web pages (e.g. author name and email).[11][35]

Information collection[edit]

Diversification and scale of personnel of Mechanical Turk allow collecting an amount of information that would be difficult outside of a crowd platform. Mechanical Turk allows Requesters to amass a large number of responses to various types of surveys, from basic demographics to academic research. Other uses include writing comments, descriptions and blog entries to websites and searching data elements or specific fields in large government and legal documents.[35]

Data processing[edit]

Companies use Mechanical Turk's crowd labor to understand and respond to different types of data. Common uses include editing and transcription of podcasts, translation, and matching search engine results.[11][35]

Research validity[edit]

The validity of research conducted with the Mechanical Turk worker pool has been questioned.[36][37] This is in large part due to the proprietary method that Mechanical Turk uses to select its workers, just like most other prominent panel providers.[38] The website may also have a bias towards young and educated people compared to other panel providers, which question the relevance of the sample if it must be representative of the general population.[39] Since the method of selection is not shared with researchers, researchers cannot know the true demographics of the pool of participants. It is unclear whether Mechanical Turk uses fiscal, political, or educational limiters in their selection process. This may invalidate any surveys or research done using the Mechanical Turk worker pool.[40][41]

Systems such as Mechanical Turk have been criticised for encouraging conformity among workers, effectively penalising those training their systems for having any independent perspectives, and potentially promoting inherent prejudices. [42]

Labor issues[edit]

Mechanical Turk has been widely criticized for its interactions with and use of labor. Computer scientist Jaron Lanier notes how the design of Mechanical Turk "allows you to think of the people as software components" that conjures "a sense of magic, as if you can just pluck results out of the cloud at an incredibly low cost".[43] While a survey done by researchers at the University of Texas, showed that the surveyed Workers were motivated by enjoyment and self-fulfillment,[44] these results may have been prejudiced by MTurk's Worker selection algorithms. A 2016 Pew Research study found that a quarter of online "gig workers" like those who work on Mechanical Turk do so because there are limited employment opportunities where they live.[45]

Monetary compensation[edit]

Because tasks are typically simple and repetitive and users are paid often only a few cents to complete them, some have criticized Mechanical Turk for exploiting and not compensating workers for the true value of the task they complete.[46] The minimum payment that Amazon allows for a task is one cent. The market for tasks is competitive and for some these tasks are their only available form of employment, particularly for the less educated. Because of the need to provide for themselves and a lack of other opportunities, many workers accept the low compensation for the completion of tasks. A study of 3.8 million tasks completed by 2,767 workers on Amazon's Mechanical Turk showed that "workers earned a median hourly wage of about $2 an hour" with 4 percent of workers earning more than $7.25 per hour. Since these workers are considered independent contractors, they are not protected by the Fair Labor Standards Act that guarantees minimum wage. By 2018, the increasing number of workers competing on the site reduced the total amount of work available. As workers search for tasks, they do not receive compensation nor do they receive additional compensation if a task takes longer than estimated by the requester.[47]


The Nation magazine said in 2014 that some Requesters had taken advantage of Workers by having them do the tasks, then rejecting their submissions in order to avoid paying them.[48]

In the Facebook–Cambridge Analytica data scandal, Mechanical Turk was one of the means of covertly gathering private information for a massive database.[49] The system paid persons a dollar or two to install a Facebook connected app and answer personal questions. The survey task, as a work for hire, was not used for a demographic or psychological research project as it might have seemed. The purpose was instead to bait the worker to reveal personal information about the worker's identity that was not already collected by Facebook or Mechanical Turk.

Labor relations[edit]

Others have criticized that the marketplace does not have the ability for the workers to negotiate with the employers. In response to the growing criticisms of payment evasion and lack of representation, a group has developed a third party platform called Turkopticon which allows workers to give feedback on their employers allowing other users to avoid potentially unscrupulous jobs and to recommend superior employers.[50][51] Another platform called Dynamo was created to allow the workers to collect anonymously and organize campaigns to better their work environment, including the Guidelines for Academic Requesters and the Dear Jeff Bezos Campaign.[52][53][54][55] Amazon has made it harder for workers to enroll in Dynamo by closing the request account that provided workers with a required code for Dynamo membership. Amazon has installed updates that prevent plugins that identify high quality human intelligence tasks from functioning on the website.[47] Additionally, there have been worker complaints that Amazon's payment system will on occasion stop working - a major issue for workers requiring daily payments.[45]

Related systems[edit]

MTurk is comparable in some respects to the now discontinued Google Answers service. However, the Mechanical Turk is a more general marketplace that can potentially help distribute any kind of work tasks all over the world. The Collaborative Human Interpreter (CHI) by Philipp Lenssen also suggested using distributed human intelligence to help computer programs perform tasks that computers cannot do well. MTurk could be used as the execution engine for the CHI.[citation needed]

The Russian search giant Yandex has developed a similar system called Toloka.[56]

See also[edit]


  1. ^ "Amazon Mechanical Turk, FAQ page". Retrieved 14 April 2017.
  2. ^ "Overview | Requester | Amazon Mechanical Turk". Retrieved 2011-11-28.
  3. ^ "Amazon Mechanical Turk".
  4. ^ Multiple sources:
  5. ^ "Artificial artificial intelligence". The Economist. 2006-06-10.
  6. ^ a b "Mturk pricing". AWS. Amazon. 2019. Retrieved 16 April 2019.
  7. ^ "AWS Developer Forums". Retrieved 14 November 2012.
  8. ^ Tamir, Dahn. "50000 Worldwide Mechanical Turk Workers". techlist. Retrieved September 17, 2014.
  9. ^ Djellel, Difallah; Filatova, Elena; Ipeirotis, Panos (2018). "Demographics and dynamics of mechanical turk workers". Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining: 135–143.
  10. ^ Alana Semuels, "The Internet Is Enabling a New Kind of Poorly Paid Hell" (January 23, 2018) Atlantic.
  11. ^ a b c "Amazon Mechanical Turk: The Digital Sweatshop" Ellen Cushing Utne Reader January–February 2013
  12. ^ Panos Ipeirotis (March 19, 2008). "Mechanical Turk: The Demographics". New York University. Retrieved 2009-07-30.
  13. ^ Panos Ipeirotis (March 16, 2009). "Turker Demographics vs Internet Demographics". New York University. Retrieved 2009-07-30.
  14. ^ Panos Ipeirotis (March 9, 2010). "The New Demographics of Mechanical Turk". New York University. Retrieved 2014-03-24.
  15. ^ "MTurk Tracker". Retrieved 1 October 2015.
  16. ^ "MTurk Tracker". Retrieved 2 May 2019.
  17. ^ a b Chandler, Jesse.; Shapiro, Danielle (2016). "Conducting Clinical Research Using Crowdsourced Convenience Samples". Annual Review of Clinical Psychology. 12: 53–81. doi:10.1146/annurev-clinpsy-021815-093623. PMID 26772208.
  18. ^ Casey, Logan; Chandler, Jesse; Levine, Adam; Proctor, Andrew; Sytolovich, Dara (2017). "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection". SAGE Open. 7 (2): 215824401771277. doi:10.1177/2158244017712774.
  19. ^ Levay, Kevin; Freese, Jeremy; Druckman, James (2016). "The Demographic and Political Composition of Mechanical Turk Samples". SAGE Open. 6: 215824401663643. doi:10.1177/2158244016636433.
  20. ^ Agley, Jon; Xiao, Yunyu; Nolan, Rachael; Golzarri-Arroyo, Lilian (2021). "Quality control questions on Amazon's Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7". Behavior Research Methods. 54 (2): 885–897. doi:10.3758/s13428-021-01665-8. ISSN 1554-3528. PMC 8344397. PMID 34357539.
  21. ^ "The New New Demographics on Mechanical Turk: Is there Still a Gender Gap?". Archived from the original on 15 March 2015. Retrieved 12 March 2015.
  22. ^ Horton, John; Chilton, Lydia (2010). The Labor Economics of Paid Crowdsourcing. Proceedings of the 11th ACM Conference on Electronic Commerce. p. 209. arXiv:1001.0627. Bibcode:2010arXiv1001.0627H. doi:10.1145/1807342.1807376. ISBN 978-1-60558-822-3. S2CID 18237602.
  23. ^ Rajpurkar, Pranav; Zhang, Jian; Lopyrev, Konstantin; Liang, Percy (2016). "SQuAD: 100,000+ Questions for Machine Comprehension of Text". arXiv:1606.05250 [cs.CL].
  24. ^ Steve Silberman (July 24, 2007). "Inside the High-Tech Search for a Silicon Valley Legend". Wired magazine. Retrieved 2007-09-16.
  25. ^ "AVweb Invites You to Join the Search for Steve Fossett". 8 September 2007. Retrieved 2011-11-28.
  26. ^ "Official Mechanical Turk Steve Fossett Results". 2007-09-24. Retrieved 2012-08-14.
  27. ^ Jim Christie (October 1, 2008). "Hikers find Steve Fossett's ID, belongings". Reuters. Archived from the original on 20 December 2008. Retrieved 2008-11-27.
  28. ^ "Let's Get Physical".
  29. ^ "Mechanical Games, online sports video for turkers | Neural".
  30. ^ "Archived copy". Archived from the original on 2015-09-06. Retrieved 2019-04-16.{{cite web}}: CS1 maint: archived copy as title (link)
  31. ^ Project:
  32. ^ "Amazon Web Services Blog: Amazon Mechanical Turk Status Update". 2005-12-06. Retrieved 2011-11-28.
  33. ^ "Documentation Archive : Amazon Web Services". Archived from the original on 2009-04-10. Retrieved 2011-11-28.
  34. ^ "Amazon Mechanical Turk API Reference".
  35. ^ a b c d "Inside Amazon's clickworker platform: How half a million people are being paid pennies to train AI". TechRepublic. 16 December 2016.
  36. ^ Feldman, Gilad. "Running Experiments with Amazon Mechanical Turk | Gilad Feldman".
  37. ^ Landers, R. N.; Behrend, T. S. (2015). "Can I Use Mechanical Turk (MTurk) for a Research Study?". Industrial and Organizational Psychology. 8 (2).
  38. ^ Lazarus, Jeffrey V. (October 6, 2020). "COVID-SCORE: A global survey to assess public perceptions of government responses to COVID-19 (COVID-SCORE-10)". PLOS ONE. 15 (10). Data Collection. Bibcode:2020PLoSO..1540011L. doi:10.1371/journal.pone.0240011. PMC 7538106. PMID 33022023.
  39. ^ Kimball, Spencer H. (2018). "Survey Data Collection; Online Panel Efficacy. A Comparative Study of Amazon MTurk and Research Now SSI/ Survey Monkey/ Opinion Access" (PDF). Business Press. Archived (PDF) from the original on 2021-10-22. Retrieved October 22, 2021.
  40. ^ "External Validity - Generalizing Results in Research".
  41. ^ "Social Research Methods - Knowledge Base - External Validity".
  42. ^ Naylor, Aliide (March 8, 2021). "Underpaid Workers Are Being Forced to Train Biased AI on Mechanical Turk".
  43. ^ Jaron Lanier (2013). Who Owns the Future?. Simon and Schuster. ISBN 978-1-4516-5497-4.
  44. ^ Buhrmester, Michael; Kwang, Tracy; Gosling, Sam (2011). "Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?". Perspectives on Psychological Science. 6 (1): 3–5. doi:10.1177/1745691610393980. PMID 26162106. S2CID 6331667.
  45. ^ a b Semuels, Alana (23 January 2018). "The Internet Is Enabling a New Kind of Poorly Paid Hell". The Atlantic. Retrieved 25 April 2019.
  46. ^ Schmidt, Florian Alexander (2013). "The Good, The Bad and the Ugly: Why Crowdsourcing Needs Ethics". 2013 International Conference on Cloud and Green Computing. Cloud and Green Computing (CGC), 2013 Third International Conference on. pp. 531–535. doi:10.1109/CGC.2013.89. ISBN 978-0-7695-5114-2. S2CID 18798641.
  47. ^ a b Semuels, Alana (January 23, 2018). "The Internet is Enabling a New Kind of Poorly Paid Hell". The Atlantic. Retrieved May 14, 2018.
  48. ^ Marvit, Moshe Z. (February 5, 2014). "How Crowdworkers Became the Ghosts in the Digital Machine".
  49. ^ New York Times (April 10, 2018). "Cambridge Analytica and the Coming Data Bust". The New York Times. Retrieved April 13, 2018.
  50. ^ Hal Hodson (February 7, 2013). "Crowdsourcing grows up as online workers unite". New Scientist. Retrieved May 21, 2015.
  51. ^ "turkopticon".
  52. ^ Mark Harris (December 3, 2014). "'Amazon's Mechanical Turk workers protest: 'I am a human being, not an algorithm''". The Guardian. Retrieved October 6, 2015.
  53. ^ Fingas, Jon (December 3, 2014). "'Amazon's Mechanical Turk workers want to be treated like humans'". Engadget. Retrieved October 6, 2015.
  54. ^ James Vincent (December 4, 2014). "Amazon's Mechanical Turkers want to be recognized as 'actual human beings". The Verge. Retrieved October 6, 2015.
  55. ^ Sarah Kessler (February 19, 2015). "WHAT DOES A UNION LOOK LIKE IN THE GIG ECONOMY?". Fast Company. Retrieved October 6, 2015.
  56. ^ "Yandex.Toloka".

Further reading[edit]

External links[edit]