Amazon Mechanical Turk
|Amazon Mechanical Turk|
|Web address||Official website|
|8,419 (January 2013[update])|
The Amazon Mechanical Turk (MTurk) is a crowdsourcing Internet marketplace that enables individuals and businesses (known as Requesters) to coordinate the use of human intelligence to perform tasks that computers are currently unable to do. It is one of the sites of Amazon Web Services. The Requesters are able to post tasks known as HITs (Human Intelligence Tasks), such as choosing the best among several photographs of a storefront, writing product descriptions, or identifying performers on music CDs. Workers (called Providers in Mechanical Turk's Terms of Service, or, more colloquially, Turkers) can then browse among existing tasks and complete them for a monetary payment set by the Requester. To place HITs, the requesting programs use an open application programming interface (API), or the more limited MTurk Requester site. Requesters are restricted to US-based entities.
- 1 Overview
- 2 History
- 3 Workers and interfaces of applicant
- 4 Noted aspects of the service
- 5 Applications
- 6 Third-party programming
- 7 Cases of uses
- 8 Related systems
- 9 Reception
- 10 See also
- 11 References
- 12 External links
The name Mechanical Turk comes from "The Turk", a chess-playing automaton of the 18th century, which was made by Wolfgang von Kempelen. It toured Europe, beating the likes of Napoleon Bonaparte and Benjamin Franklin. It was later revealed that this "machine" was not an automaton at all, but was in fact a chess master hidden in a special compartment controlling its operations. Likewise, the Mechanical Turk web service allows humans to help the machines of today perform tasks for which they are not suited.
Workers set their own hours and are not under any obligation to accept any work they do not wish to do. Because workers are paid as contractors rather than employees, requesters do not have to file forms for, nor pay payroll taxes, and they avoid laws regarding minimum wage, overtime, and workers compensation. Workers, though, must report their income as self-employment income. The average wage for the multiple microtasks assigned, if they are done quickly, is about one dollar an hour, with each task averaging a few cents.
Requesters can ask that Workers fulfill Qualifications before engaging a task, and they can set up a test in order to verify the Qualification. They can also accept or reject the result sent by the Worker, which reflects on the Worker's reputation. Workers can have a postal address anywhere in the world. Payments for completing tasks can be redeemed on Amazon.com via gift certificate (gift certificates are the only payment option available to international workers, apart from India) or be later transferred to a Worker's U.S. bank account. Requesters pay Amazon a 10% commission on the price of successfully completed HITs.
According to a survey conducted in 2008 through one MTurk HIT, Turkers are primarily located in the United States with demographics generally similar to the overall Internet population in the US.
The same author carried out a second survey in 2010 (after the introduction of cash payments for Indian workers), giving new and updated results on the demographics of workers.
A more recent study reports Worker demographics on over 30,000 Workers across 75 studies that have been conducted since 2013.
The service was initially invented by Peter Cohen for Amazon's internal use, to find duplicates among its web pages describing products.
MTurk was launched publicly on November 2, 2005. Following its launch, the Mechanical Turk user base grew quickly. In early- to mid-November 2005, there were tens of thousands of HITs, all of them uploaded to the system by Amazon itself for some of its internal tasks that required human intelligence. Most of these were related to music CD items. HIT types have expanded to include transcribing, rating, image tagging, surveys, and writing.
In March 2007, there were reportedly more than 100,000 workers in over 100 countries. This increased to over 500,000 workers from over 190 countries in January 2011. In the same year, Techlist published an interactive map pinpointing the locations of 50,000 of their MTurk workers around the world.
Workers and interfaces of applicant
Users visiting Mechanical Turk website can become either a worker or an applicant. Workers have access to a control panel that displays three tabs: total earnings, the state of the HIT and the total HIT.
- Total earnings: displays the total earnings that a worker has received from the realization of human intelligence tasks, the gains made by bonus and the sum of these two.
- State of the HIT: displays a list of daily activities and the daily income, along with the number of visits that were submitted, approved, rejected or waiting for that day.
- Total HIT: displays information about HIT which have been accepted or are in process (including the percentage of successes that occurred, returned or abandoned and the percentage of hits that were approved, rejected or pending those presented).
Applicants (companies or independent developers) that need HIT realisation to be made can use the Amazon Mechanical Turk API to access thousands of employees on demand, high quality, low cost and all over the world and then integrate programmatically the results of that work directly on their business processes and systems. Mechanical Turk allows applicants to achieve their goals more quickly and at lower cost than was previously possible. When applicants send their HIT, they should specify how much time the workers have to complete the work, how many people can develop each task, how much are they paying for the work and the specific details about the job they want to be completed.
Noted aspects of the service
Personnel on demand
Amazon Mechanical Turk provides access to a crowd-sourced market of workers that can help to complete work on an as-needed basis. For work that does not require significant task-specific training, this can contrast with the traditional costs of hiring and management of temporary staff. For users, it also allows them to find ways to contribute to a variety of different possible tasks.
Amazon Mechanical Turk allows more than one user to send a response to the same HIT. When a specific number of users give the same answer, the HIT is automatically approved. The payments are made only to those that are considered successful. If the result is not adequate, the job is rejected and the Requester is not required to pay.
Users are free to work on tasks that they find most interesting, those they like to complete or the best paid. Requesters are allowed to define the payments based on the desired balance of performance and cost-efficiency. Payments are made in cooperation with Amazon Payments.
Amazon Mechanical Turk allows for qualifying users before they work in their tasks using rapid tests. The qualifications can be a series of questions, performing tasks or request users to have historically responded to a minimum percentage of their HIT sent correctly.
Missing persons searches
Since 2007, the service has been used to search for prominent missing individuals. It was first suggested during the search for James Kim, but his body was found before any technical progress was made. That summer, computer scientist Jim Gray disappeared on his yacht and Amazon's Werner Vogels, a personal friend, made arrangements for DigitalGlobe, which provides satellite data for Google Maps and Google Earth, to put recent photography of the Farallon Islands on Mechanical Turk. A front-page story on Digg attracted 12,000 searchers who worked with imaging professionals on the same data. The search was unsuccessful.
In September 2007, a similar arrangement was repeated in the search for aviator Steve Fossett. Satellite data was divided into 85 squared meter sections, and Mechanical Turk users were asked to flag images with "foreign objects" that might be a crash site or other evidence that should be examined more closely. This search was also unsuccessful, partly due to the limited search area. The satellite imagery was mostly within a 50-mile radius. The crash site was eventually found by hikers about a year later, 65 miles away.
Social science experiments
Beginning in 2010, numerous researchers have explored the viability of Mechanical Turk to recruit subjects of social science experiments. In general, researchers found that while the sample of respondents obtained through Mechanical Turk does not perfectly match characteristics of the U.S. population, it doesn't present a wildly inaccurate view either. They determined that the service works best for recruiting a diverse sample; it is less successful with studies that require more precisely defined populations or that require a representative sample of the population as a whole. Overall, the US MTurk population is mostly female and white, and is somewhat younger and more educated than the US population overall. Data collected on HITs conducted since 2013 show that the US population is no longer predominantly female, and that Workers are currently slightly more likely to be male.
The cost of MTurk was considerably lower than other means of conducting surveys, with workers willing to complete tasks for less than half the US minimum wage.
Artistic and educational research
In addition to growing interest from the social sciences, MTurk has also been used as both a tool for artistic and educational exploration. Artist Aaron Koblin has made use of MTurk's crowdsourcing ability to create a number of collaborative artistic works such as The Sheep Market and Ten Thousand Cents  which combined thousands of individual drawings of a US$100 bill. The work functions as a sort of reverse exquisite corpse drawing.
Inspired by Koblin's collaborative artworks a Concordia University graduate research student turned to MTurk to see if the crowdsourcing technology could also be used for educational research. Scott McMaster conducted two pilot projects which used HITs to request drawings; but unlike Koblin's work, the Turkers knew exactly what the drawings were being used for. The HITs required participants to visually represent sets of words in drawings and fill out a short demographic survey. Although the research would be considered in its infancy, McMaster made several findings which suggest that a globalizing effect is taking place within visual cultural representations. It is a published instance of this type of online research into visual culture.
Programmers have developed various browser extensions and scripts designed to simplify the process of completing HITs. According to the Amazon Web Services Blog, however, Amazon appears to disapprove of the ones that completely automate the process and preclude the human element. Accounts using so-called automated bots have been banned.
Amazon makes available an application programming interface (API) to give users another access point into the MTurk system. The MTurk API lets a programmer submit HITs to MTurk, retrieve completed work, and approve or reject that work. Web sites and web services can use the API to integrate MTurk work into other web applications, providing users with alternatives to the interface Amazon has built for these functions.
Cases of uses
Processing photos / videos
Amazon Mechanical Turk is ideal for processing images. Although it is difficult for computers, it is easy for people to do tasks such as:
- Label the objects found in an image to make the search easier
- Select the best picture of a group of pictures (best represents the product)
- Audit images uploaded by users for inappropriate content
- Classify objects found in images by satellite
Data cleaning / verification
Companies with large online catalogs use Mechanical Turk to identify duplicates and verify details of items entries. Some examples are:
- Deduplication listings yellow pages directories
- Identify duplicate online product catalogs
- Check restaurant details (phone number, hours, etc.)
Diversification and scale of personal of Mechanical Turk allow collecting more information that would be almost impossible, for example:
- Allows users to ask questions, either from a computer or mobile device, on any subject, so that applicants can give results on these questions
- Fill surveys data on various themes
- Write comments, descriptions and blog entries to websites
- Search data elements or specific fields in large government and legal documents
Companies use the potential of the template Mechanical Turk to understand and respond intelligently to different types of data:
- Editing and transcription of podcasts
- Human translation service
- Classification accuracy of the results of a search engine
Amazon coined the term artificial artificial intelligence for processes outsourcing some parts of a computer program to humans, for those tasks carried out much faster by humans than computers. Jeff Bezos was responsible for the concept that led to Amazon's Mechanical Turk being developed to realize this process.
MTurk is comparable in some respects to the now discontinued Google Answers service. However, the Mechanical Turk is a more general marketplace that can potentially help distribute any kind of work tasks all over the world. The Collaborative Human Interpreter (CHI) by Philipp Lenssen also suggested using distributed human intelligence to help computer programs perform tasks that computers cannot do well. MTurk could be used as the execution engine for the CHI.
Because HITs are typically simple, repetitive tasks and users are paid often only a few cents to complete them, some have criticized Mechanical Turk for exploiting and not compensating workers for the true value of the task they complete.  However, in a psychological study done by the University of Texas, evidence showed that many of the workers did not complete the task for monetary compensation, and instead did the work for enjoyment and self-fulfillment. The Nation magazine said in 2014 that some requesters had taken advantage of workers by having them do the tasks, then rejecting their submissions in order to avoid paying them. Others have criticized that the marketplace does not have the ability for the workers to negotiate with the employers. In response to the growing criticisms of payment evasion and lack of representation, a group has developed a third party platform called Turkopticon which allows workers to give feedback on their employers allowing other users to avoid potentially shady jobs and to recommend superior employers.
- "mturk.com Site Info". Alexa Internet. Retrieved 2013-01-26.
- "Overview | Requester | Amazon Mechanical Turk". Requester.mturk.com. Retrieved 2011-11-28.
- "Can international Requesters use Amazon Mechanical Turk to get tasks completed". mturk.com. Retrieved 2012-06-13.
- "Amazon Mechanical Turk: The Digital Sweatshop" Ellen Cushing Utne Reader January–February 2013:
- "Mturk pricing". AWS. Amazon. 2014. Retrieved 21 July 2014.
- Panos Ipeirotis (March 19, 2008). "Mechanical Turk: The Demographics". New York University. Retrieved 2009-07-30.
- Panos Ipeirotis (March 16, 2009). "Turker Demographics vs Internet Demographics". New York University. Retrieved 2009-07-30.
- Panos Ipeirotis (March 9, 2010). "The New Demographics of Mechanical Turk". New York University. Retrieved 2014-03-24.
- "The New New Demographics on Mechanical Turk: Is there Still a Gender Gap?". TurkPrime.com. Retrieved 12 March 2015.
- "AWS Developer Forums". Retrieved 14 November 2012.
- Tamir, Dahn. "50000 Worldwide Mechanical Turk Workers". techlist. Retrieved September 17, 2014.
- Steve Silberman (July 24, 2007). "Inside the High-Tech Search for a Silicon Valley Legend". Wired magazine. Retrieved 2007-09-16.
- "AVweb Invites You to Join the Search for Steve Fossett". Avweb.com. Retrieved 2011-11-28.
- "Official Mechanical Turk Steve Fossett Results". Amazon.com. 2007-09-24. Retrieved 2012-08-14.
- Jim Christie (October 1, 2008). "Hikers find Steve Fossett's ID, belongings". Reuters. Archived from the original on 20 December 2008. Retrieved 2008-11-27.
- "Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk". JournalistsResource.org, retrieved June 18, 2012
- Paolacci, Gabriele.; Chandler, Jesse; Ipeirotis, Panos (2010). "Running Experiments on Amazon Mechanical Turk" (PDF). Judgment and Decision Making.
- Buhrmester, Michael; Kwang, Tracy; Gosling, Sam (2011). "Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?". Perspectives on Psychological Science. doi:10.1177/1745691610393980.
- Berinsky, Adam J.; Huber, Gregory A.; Lenz, Gabriel S. (2012). "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk". Political Analysis. doi:10.1093/pan/mpr057.
- Horton, John; Chilton, Lydia (2010). "The Labor Economics of Paid Crowdsourcing" (PDF). Proceedings of the 11th ACM conference on Electronic commerce. doi:10.1145/1807342.1807376.
- Ten Thousand Cents - Project: http://www.tenthousandcents.com/top.html
- Koblin, Aaron: http://www.aaronkoblin.com/work.html
- McMaster, S. (2012). New Approaches to Image-based Research and Visual Literacy. In Avgerinou, Chandler, Search and Terzic (Eds.), New Horizons in Visual Literacy: Selected Readings of the International Visual Literacy Association (122-132). Siauliai, Lithuania: SMC Scientia Educologica: http://concordia.academia.edu/SCOTTMCMASTER
- "Amazon Web Services Blog: Amazon Mechanical Turk Status Update". Aws.typepad.com. 2005-12-06. Retrieved 2011-11-28.
- "Documentation Archive : Amazon Web Services". Developer.amazonwebservices.com. Retrieved 2011-11-28.
- "Artificial artificial intelligence". The Economist. 2006-06-10.
- F.A. Schmidt (September 30, 2013). "'The Good, The Bad and the Ugly: Why Crowdsourcing Needs Ethics'". IEEE Xplore. Retrieved May 21, 2015.
- Moshe Z. Marvit, "How Crowdworkers Became the Ghosts in the Digital Machine," The Nation, February 24, 2014, screen 4
- Hal Hodson (February 7, 2013). "’Crowdsourcing grows up as online workers unite’". NewScientist. Retrieved May 21, 2015.
- Official website
- Wired Magazine story about "Crowdsourcing," June 2006.
- Business Week article on Mechanical Turk by Rob Hof, November 4, 2005.
- New York Times article on Mechanical Turk by Jason Pontin, March 25, 2007.
- Salon.com article on Mechanical Turk by Katharine Mieszkowski, July 24, 2006.
- Technology Review article on Mechanical Turk, "How Mechanical Turk is Broken," by Christopher Mims, January 3, 2010.
- Requester Best Practices Guide, Updated June 2011.