Crowdsourcing

From Wikipedia, the free encyclopedia
Jump to: navigation, search
It has been suggested that Digital sweatshop and Microwork be merged into this article or section. (Discuss) Proposed since March 2012.
Editing Hoxne Hoard at the British Museum.ogv
Wikipedians and British Museum curators collaborate on the article Hoxne Hoard in June 2010.

Crowdsourcing is a distributed problem-solving and production process that involves outsourcing tasks to a network of people, also known as the crowd. This process can occur both online and offline.[1] The difference between crowdsourcing and ordinary outsourcing is that a task or problem is outsourced to an undefined public rather than a specific other body.

Crowdsourcing is related to, but not the same as, human-based computation, which refers to the ways in which humans and computers can work together to solve problems. These two methods can be used together to accomplish tasks.[2]

Contents

[edit] Overview

 This section of the article is too long to read comfortably, and needs subsections. Please format the article according to the guidelines laid out in the Manual of Style. (March 2012)

Crowdsourcing is a distributed problem-solving and production model. In the classic use of the term, problems are broadcast to an unknown group of solvers in the form of an open call for solutions. Users—also known as the crowd—submit solutions. Solutions are then owned by the entity that broadcast the problem in the first place—the crowdsourcer. The contributor of the solution is, in some cases, compensated either monetarily, with prizes, or with recognition. In other cases, the only rewards may be kudos or intellectual satisfaction. Crowdsourcing may produce solutions from amateurs or volunteers working in their spare time, or from experts or small businesses which were unknown to the initiating organization.[1]

As crowdsourcing is a broad, process-based term, there are numerous examples of ways in which this technique has been adopted to accomplish tasks related to medicine, business, research, and art. Tools used to crowdsource tasks serve a variety of functions, including breaking down tasks and helping crowdsourcers find people to contribute to their tasks. There are hundreds of crowdsourcing platforms available online - Amazon’s Mechanical Turk is a notable example because of its early creation in 2005 and its sustained popularity. Regardless of individual features, the goal of these systems remains the same – to distribute tasks to the crowd.

Those who utilize crowdsourcing services, also known as crowdsourcers, are motivated by the benefits of crowdsourcing, which are that you can gather large numbers of solutions or information and that it is relative inexpensive to obtain this work. Users are are motivated to contribute to crowdsourced tasks by both intrinsic motivations, such as social contact and passing the time, and extrinsic motivations, such as financial gain.

It is important to note that while there are many benefits of crowdsourcing, is not without criticism. Indeed, both within industry and academia, people have questioned the ethical validity of providing no compensation or small amounts of compensation to members of the crowd that contribute to tasks. In addition, others have suggested that the crowd is not equally suited to accomplish all types of tasks.

[edit] Etymology

The term "crowdsourcing" is a portmanteau of "crowd" and "outsourcing," coined by Jeff Howe in a June 2006 Wired magazine article "The Rise of Crowdsourcing".[3]

[edit] Projects

Crowdsourcing systems are used to accomplish a variety of tasks. For example, the crowd may be invited to develop a new technology, carry out a design task (also known as community-based design [4] or distributed participatory design), refine or carry out the steps of an algorithm (see human-based computation), or help capture, systematize or analyze large amounts of data (see also citizen science).

[edit] Historical Examples

 This section of the article is too long to read comfortably, and needs subsections. Please format the article according to the guidelines laid out in the Manual of Style. (March 2012)

Long before modern crowdsourcing systems were developed, there are a number of notable examples of projects that utilized distributed people to help accomplish tasks.

The Oxford English Dictionary (OED) may provide one of the earliest examples of crowdsourcing. An open call was made to the community for contributions by volunteers to index all words in the English language and example quotations for each and every one of their usages. In the 70 year project, they received over 6 million submissions. The making of the OED is detailed in The Surgeon of Crowthorne by Simon Winchester. in 1994, Northeast Consulting compiled a database of trends in the marketplace. This database was collected from numerous sources, offering an example of early crowdsourcing.[5]

Similarly, crowdsourcing has often been used in the past as a competition in order to discover a solution. The French government proposed several of these competitions, often rewarded with Montyon Prizes, created for poor Frenchmen who had done virtuous acts[6], including the Leblanc process, also known as the Alkali Prize, where a reward was provided for separating the salt from the alkali, and the Fourneyron's Turbine, when the first hydraulic commercial turbine was developed. [7] In response to a challenge from the French government, Nicholas Appert won a prize for inventing a new way of food preservation that involved sealing food in air-tight jars [8] The British government provided a similar reward to find an easy way to determine a ship’s longitude in the The Longitude Prize. During the Great Depression, out-of-work clerks tabulated higher mathematical functions in the Mathematical Tables Project as an outreach project.[9]

[edit] Modern Crowdsourcing

Today, crowdsourcing has transferred mainly to the web. The web provides a particularly good venue for crowdsourcing since individuals tend to be more open in web-based projects where they are not being physically judged or scrutinized and thus can feel more comfortable sharing.[10] This ultimately allows for well-designed artistic projects because individuals are less conscious, or maybe even less aware, of scrutiny towards their work. In an online atmosphere more attention is given to the project rather than communication with other individuals.

Within the umbrella of crowdsourcing, there are several different types which can be used depending on the project. Some of these web-based crowdsourcing efforts include crowdvoting, crowd wisdom, microwork, and inducement prize contests.

[edit] Crowdvoting

Commerical websites employ crowdsourcing in order to gain community opinions, also known as crowdvoting. Threadless.com crowdsources its t-shirt designs by having users provide designs and then vote on the ones they like, which are then printed and available for purchase. The company itself has less than 20 employees, yet there are thousands of members who provide designs and vote on them, making the website’s products truly created and selected by the crowd, rather than the company.[11] Another example is Minted.com, which crowdsources designs for paper goods, like cards, stationary, and calendars in order to sell the most popular ones.[12]

[edit] Crowdwisdom

 This section of the article is too long to read comfortably, and needs subsections. Please format the article according to the guidelines laid out in the Manual of Style. (March 2012)

Crowdwisdom is another type of crowdsourcing that collects large amounts of information and aggregates it. One major example is Wikipedia itself. All over the world, hundreds of thousands of users are contributing information on almost 4 million topics in order to create complete articles. Without these multitudes of users, this site would not be able to be as expansive.[13]

The 2011 documentary Life In a Day used crowdsourcing to collect video footage from people worldwide. Contributors from 192 countries submitted 4,500 hours of video, primarily through YouTube uploads, which the filmmakers edited to 97 minutes in order to create a cohesive documentary.

iStockPhoto provides a platform for people to upload photos and purchase them for low prices. In order to become a member of the site, three photos must be submitted for judging by the iStockPhoto staff. Clients can purchase photos through credits and photographers get a small profit. The many members of the site create a one-stop shop for retrieving photos that would not be as effective if they were across many sites.[11]

Research and development provides many opportunities for crowdsourcing people’s thoughts and ideas. The company InnoCentive is a crowdsourcing platform for corporate research and development, where difficult scientific problems are posted for crowds of solvers to discover the answer and win a cash prize, which can range from $10,000 to $100,000 per challenge.[11]

Harvard Tuberculosis Lab teamed with CrowdFlower to help identify drug resistant TB cells in mouse cortex slides. If they had not used crowdsourcing, the project would have been stalled considerably because they did not have enough people on their own to look at all the images.[14]

In February 2012, a stock picking game called Ticker Picker Pro was launched, using crowdsourcing technologies in order to create a hedge fund that would buy and sell stocks based on the ideas coming out of the game. To date no official results have been released as to the reliability of their proprietary technologies.[15]

[edit] Microwork

Microwork is a crowdsourcing platform where users do small tasks that computers are not good at for very low amounts of money. The most popular example is Amazon’s Mechanical Turk, which has created many different projects for users to participate in, where each task requires very little time and offers a very small amount in payment.[1] One of Mechanical Turk’s more recent projects was when users searched satellite images for images of a boat in order to find lost researcher Jim Gray.[16]

[edit] Inducement Prize Contests

Web-based idea competitions, or inducement prize contests often consist of generic ideas, cash prizes, and an Internet-based platform to facilitate easy idea generation and discussion. Examples of these competitions include events like IBM’s 2006 “Innovation Jam”, attended by over 140,000 international participants and yielding around 46,000 ideas.[17] [18]

Another example of competition-based crowdsourcing is the 2009 DARPA experiment, where DARPA placed 10 balloon markers across the United States and challenged teams to compete to be the first to report the location of all the balloons. A collaboration of efforts was required to complete the challenge quickly and in addition to the competitive motivation of the contest as a whole, the winning team (MIT, in less than nine hours) established its own "collaborapetitive" environment to generate participation in their team.[19]

[edit] Tools

In order to foster this community environment for completing tasks, a variety of tools exist, particularly on the Internet. These tools facilitate the gathering of information effectively and can be divided into 2 groups: explicit and implicit. Explicit crowdsourcing lets users work together to evaluate, share, and build different specific tasks, while implicit crowdsourcing means that users solve a problem as a side effect of something else they are doing.

With explicit crowdsourcing, users can evaluate particular items like books or webpages, whereas sharing involves posting products or items, particularly on sites like YouTube and Napster. Users can also build artifacts, such as Wikipedia and open-source sites like Linux.[16] Another type of tool is the micro-task market, where small tasks can be selected by users and completed for a small monetary reward. The most popular example of this tool is Amazon’s Mechanical Turk, which was originally created to have humans do simple tasks computers would find difficult.The Chinese versions of this, commonly called Witkeys, are similar and include such sites as Taskcn.com and k68.cn. These micro-task markets differ from open-question forums because they provide small amounts of payment for task completion. Taskcn.com has over 3000 different tasks posted in 7 different categories in order to cover many different topics, from design to programming to strategy planning. When choosing tasks, since oftentimes only certain users “win”, users learn to submit later and pick less popular tasks in order to increase the likelihood of getting their work selected.[20] Sites that build artifacts, such as Wikipedia itself, are also explicit crowdsourcing, where people provide information and edit other people’s work.

Implicit crowdsourcing can take two forms: standalone and piggyback. Standalone allows people to solve problems as a side effect of the task they are actually doing, whereas piggyback takes users’ information from a third-party website. In terms of standalone implicit crowdsourcing, an example is the ESP game, where users guess what images are and then these labels are used to tag Google images. reCAPTCHA asks people to solve captchas in order to prove they are human, and then provides captchas from old books that cannot be deciphered by computers in order to try and digitize them for the web. This task is simple for humans, but would be incredibly difficult for computers. Piggyback crowdsourcing can be seen most frequently by websites such as Google that mine one’s search history and websites in order to discover keywords for ads, spelling corrections, and finding synonyms. In this way, users are unintentionally helping to modify existing systems, such as Google’s ad words.[21]

[edit] Crowdsourcers

There are a number of motivations for businesses to use crowdsourcing to accomplish tasks, find solutions for problems, or to gather information. These include the ability to offload peak demand, access cheap labor and information, generate better results, access a wider array of talent than might be present in one organization, and undertake problems that would have been too difficult to solve internally.[22] Crowdsourcing allows businesses to submit problems in which contributors can work on, such as problems in science, manufacturing, biotech, and medicine, with monetary rewards for successful solutions. Although it can be difficult to crowdsource complicated tasks, simple work tasks can be crowdsourced cheaply and effectively.

Crowdsourcing also has the potential to be a problem-solving mechanism for government and nonprofit use. Urban and transit planning are prime areas for crowdsourcing. One project to test crowdsourcing's public participation process for transit planning in Salt Lake City has been underway from 2008 to 2009, funded by a U.S. Federal Transit Administration grant.[23] Another notable application of crowdsourcing to government problem solving is the Peer to Patent Community Patent Review project for the U.S. Patent and Trademark Office.[24]

Researchers have used crowdsourcing systems, in particular Mechanical Turk, to aid with research projects by crowdsourcing aspects of the research process such as data collection, parsing, and evaluation. Notable examples include using the crowd to create speech and language databases [25] [26], and using the crowd to conduct user studies [21]. Crowdsourcing systems provide these researchers with the ability to gather large amount of data. Additionally, using crowdsourcing, researchers can collect data from populations and demographics they may not have had access to locally, but that improve the validity and value of their work. [27]

Artists have also utilized crowdsourcing systems. In his project the Sheep Market, Aaron Koblin used Mechanical Turk to collect 10,000 drawings of sheep from contributors around the world.[28]. Sam Brown (artist) leverages the crowd by asking visitors of his website explodingdog to send him sentences that he uses as inspirations for paintings.[29] Art curator Andrea Grover argues that individuals tend to be more open in crowdsourced projects because they are not being physically judged or scrutinized.[10] As with other crowdsourcers, artists use crowdsourcing systems to generate and collect data. The crowd also can be used to provide inspiration for an artist’s work and also to collect financial support for that work. [30]

[edit] The Crowd

[edit] Demographics

The crowd is an umbrella term for people who contribute to crowdsourcing efforts. Though it is sometimes difficult to gather data about the demographics of the crowd, a study by Ross et al surveyed the demographics of a sample of the more than 400,000 registered crowdworkers using Amazon Mechanical Turk to complete tasks for pay. While a previous study in 2008 by Ipeirotis found that users at that time were primarily American, young, female, and well-educated, with 40% having incomes >$40,000/yr, in 2009 Ross found a very different population. By Nov. 2009, 36% of the surveyed Mechanical Turk workforce was Indian. ⅔ of Indian workers were male, and 66% had at least a Bachelor’s degree. ⅔ had annual incomes less than $10,000/yr, with 27% sometimes or always depending on income from Mechanical Turk to make ends meet. [31]

The average US user of Mechanical Turk earned $2.30 per hour for tasks in 2009, versus $1.58 for the average Indian worker. While the majority of users worked less than 5 hours per week, 18% worked 15 hours per week or more. This is less than minimum wage in either country, which Ross suggests raises ethical questions for researchers who use crowdsourcing.

The demographics of http://microworkers.com/ differ from Mechanical Turk in that the US and India together account for only 25% of workers. 197 countries are represented among users, with Indonesia (18%) and Bangladesh (17%) contributing the largest share. However, 28% of employers are from the US. [32]

[edit] Motivations

Kaufmann and Schulze suggest that there are both intrinsic and extrinsic motivations that cause people to contribute to crowdsourced tasks, and that these factors influence different types of contributors. For example, students and people employed full-time rate Human Capital Advancement as less important than part-time workers do, while woman rate Social Contact as more important than men do. [33]

Intrinsic motivations are broken down into two categories, enjoyment-based and community-based motivations. Enjoyment-based motivations refer to motivations related to the fun and enjoyment that the contributor experiences through their participation. These motivations include: skill variety, task identity, task autonomy, direct feedback from the job, and pastime. Community-based motivations refer to motivations related to community participation, and include community identification and social contact.

Extrinsic motivations are broken down into three categories, immediate payoffs, delayed payoffs, and social motivations. Immediate payoffs, through monetary payment, are the immediately received compensations given to those who complete tasks. Delayed payoffs are benefits that can be used to generate future advantages, such as training skills and being noticed by potential employers. Social motivations are the rewards of behaving pro-socially, such as altruistic motivations. Chandler and Kapelner found that US users of the Amazon Mechanical Turk were more likely to complete a task when told they were going to “help researchers identify tumor cells,” than when they were not told the purpose of their task. However, of those who completed the task, quality of output did not depend on the framing of the task. [33]

Another form of social motivation is prestige or status. The International Children’s Digital Library recruits volunteers to translate and review books. Because all translators receive public acknowledgment for their contribution, Kaufman and Schulz cite this as a reputation-based strategy to motivate individuals who want to be associated with institutions that have prestige. The Amazon Mechanical Turk uses reputation as a motivator in a different sense, as a form of quality control. Crowdworkers who frequently complete tasks in ways judged to be inadequate can be denied access to future tasks, providing motivation to produce high-quality work. [2]

[edit] Criticisms

There are two major categories of criticisms about crowdsourcing, (1) the value and impact of the work received from the crowd and (2) the ethical implications of low wages paid to crowdworkers. Most of these criticisms are directed towards crowdsourcing systems that provide extrinsic monetary rewards to contributors, though some apply more generally to all crowdsourcing systems.

[edit] Concerns for Crowdsourcers

Concerns for crowdsources include:

Susceptibility to faulty results caused by targeted, malicious work efforts. Since crowdworkers completing microtasks are paid per task, there is often a financial incentive to complete tasks quickly rather than well. Verifying responses is time consuming, and so requesters often depend on having multiple workers complete the same task to correct errors. However, having each task completed multiple times increases time and monetary costs. [34]

Crowdworkers are a nonrandom sample of the population. Many researchers use crowdsourcing in order to quickly and cheaply conduct studies with larger sample sizes than would be otherwise achievable. However, due to low worker pay, participant pools are skewed towards poor users in developing countries.[35][34]

Ethical concerns. Because crowdworkers are considered independent contractors rather than employees, they are not guaranteed a minimum wage. In practice, workers using the Amazon Mechanical Turk generally earn less than the minimum wage, even in India. [31] [36]Some researchers considering using Mechanical Turk to get participants for studies have argued that this may be unethical.[37][27]

Increased likelihood that a crowdsourced project will fail due to lack of monetary motivation or too few participants. Crowdsourcing markets are not a first-in-first-out queue. Tasks that are not completed quickly may be forgotten, buried by filters and search procedures so that workers do not see them. This results in a long tail power law distribution of completion times. [38] Additionally, low-paying research studies online have higher rates of attrition, with participants not completing the study once started.[27] Even when tasks are completed, crowdsourcing doesn't always produce quality results. When Facebook began its localization program in 2008, it encountered criticism for the low quality of its crowdsourced translations.[39]

[edit] Concerns for the Crowd

Concerns for members of the crowd include:

Below-market wages. The average US user of Mechanical Turk earned $2.30 per hour for tasks in 2009, versus $1.58 for the average Indian worker. While the majority of users worked less than 5 hours per week, 18% worked 15 hours per week or more, and 27% of Indian users said income from Mechanical Turk is sometimes or always necessary for them to make ends meet. This is less than minimum wage in either country, which Ross et al. suggest raises ethical questions for researchers who use crowdsourcing.[31] When Facebook began its localization program in 2008, it received criticism for using crowdsourcing to obtain free labor.[39]

No written contracts, non-disclosure agreements, or employee agreements or agreeable terms with crowdsourced employees. For users of the Amazon Mechanical Turk, this means that requestors have final say over whether users’ work is acceptable; if not, they will not be paid. [40]

Difficulties in collaboration of crowd members, especially in the context of competitive crowd sourcing. Crowdsourcing site InnoCentive allows organizations to solicit solutions to scientific and technological problems; only 10.6% of respondents report working in a team on their submission. [41]


[edit] Ubiquitious human computing

Ubiquitous human computing is the phenomenon of dis-aggregating a task into component pieces and then parceling them out around the world. Perhaps the best-known example is Amazon Mechanical Turk, where simple tasks that cannot be done by a computer—for example, labeling images—are outsourced to anyone with an internet connection for a low cost.

[edit] See Also

[edit] References

  1. ^ a b c Jeff Howe (2006). "The Rise of Crowdsourcing". Wired. http://www.wired.com/wired/archive/14.06/crowds.html. 
  2. ^ a b Quinn; Bederson (2010). "Human Computation: A Survey and Taxonomy of a Growing Field". CHI 2011. http://alexquinn.org/papers/Human%20Computation,%20A%20Survey%20and%20Taxonomy%20of%20a%20Growing%20Field%20(CHI%202011).pdf. 
  3. ^ David Whitford (March 22, 2007). "Hired Guns on the Cheap". Fortune Small Business. http://money.cnn.com/magazines/fsb/fsb_archive/2007/03/01/8402019/index.htm. Retrieved August 7, 2008. 
  4. ^ David Whitford (January 8, 2010). "Crowd Sourcing Turns Business On Its Head". CNN. http://www.npr.org/templates/story/story.php?storyId=93495217. Retrieved February 27, 2012. 
  5. ^ "Looking Forward - Emerging and Declining Networks for 2009". 21st Century Organization Blog. February 1, 2009. http://c21org.typepad.com/21st_century_organization/2009/02/looking-forward---emerging-and-declining-networks-for-2009.html. Retrieved November 11, 2011. 
  6. ^ "Antoine-Jean-Baptiste-Robert Auget, Baron de Montyon". New Advent. http://www.newadvent.org/cathen/10552a.htm. Retrieved February 25, 2012. 
  7. ^ "It Was All About Alkali". Chemistry Chronicles. http://pubs.acs.org/subscribe/archive/tcaw/11/i01/html/01chemchron.html. Retrieved February 25, 2012. 
  8. ^ "Nicolas Appert". John Blamire. http://www.brooklyn.cuny.edu/bc/ahp/MBG/MBG4/Appert.html. Retrieved February 25, 2012. 
  9. ^ "9 Examples of Crowdsourcing, Before ‘Crowdsourcing’ Existed". MemeBurn. http://memeburn.com/2011/09/9-examples-of-crowdsourcing-before-%E2%80%98crowdsourcing%E2%80%99-existed/. Retrieved February 25, 2012. 
  10. ^ a b DeVun, Leah (November 19, 2009). "Looking at how crowds produce and present art.". Wired News. http://www.wired.com/techbiz/media/news/2007/07/crowd_captain?currentPage=all. 
  11. ^ a b c Brabham, Daren (2008), "Crowdsourcing as a Model for Problem Solving: An Introduction and Cases", Convergence: The International Journal of Research into New Media Technological Studies 14 (1): 75-90, http://www.clickadvisor.com/downloads/Brabham_Crowdsourcing_Problem_Solving.pdf 
  12. ^ "Minted". http://www.minted.com/about-us/. 
  13. ^ "Wikipedia Statistics". Wikipedia. http://en.wikipedia.org/wiki/Special:Statistics. 
  14. ^ Burke, Adrienne (October 16, 2011). "Crowdsourcing Scientific Progress: How Crowdflower's Hordes Help Harvard Researchers Study TB". Forbes. http://www.forbes.com/sites/techonomy/2011/10/26/crowdsourcing-scientific-progress-how-crowdflowers-hordes-help-harvard-researchers-study-tb/. 
  15. ^ Rulison, Larry (February 14, 2012). "A Winning App? They Hope So". Times Union (Albany). http://www.timesunion.com/business/article/A-winning-app-They-hope-so-3323905.php. 
  16. ^ a b Doan, A; Ramarkrishnan, R; Halevy, A (2011), "Crowdsourcing Systems on the World Wide Web", Communications of the ACM 54 (4): 86-96, http://delivery.acm.org/10.1145/1930000/1924442/p86-doan.pdf?ip=71.182.229.123&acc=OPEN&CFID=87199630&CFTOKEN=25539817&__acm__=1330408287_b904f7bffdd69c8ad85d83c619e9a3c1 
  17. ^ Leimeister, J.M.; Huber, M; Bretschneider, U; Krcmar, H (2009), "Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition", Journal of Management Information Systems 26 (1): 197-224, http://portal.acm.org/citation.cfm?id=1653890 
  18. ^ Ebner, W; Leimeister, J; Krcmar, H (2009), "Community Engineering for Innovations: The Ideas Competition as a method to nurture a Virtual Community for Innovations", R&D Management 39 (4): 342-356, http://www3.interscience.wiley.com/search/allsearch?mode=viewselected&product=journal&ID=122535413&view_selected.x=67&view_selected.y=8 
  19. ^ "DARPA Network Challenge". DARPA Network Challenge. https://networkchallenge.darpa.mil/default.aspx. Retrieved November 28, 2011. 
  20. ^ Yang, J; Adamic, L; Ackerman, M (2008), "Crowdsourcing and Knowledge Sharing: Strategic User Behavior on Taskcn", Proceedings of the 9th ACM Conference on Electronic Commerce, http://www-personal.umich.edu/~ladamic/papers/taskcn/EC2008Witkey.pdf 
  21. ^ a b Kittur, A; Chi, E.H.; Sun, B (2008), "Crowdsourcing user studies with Mechanical Turk", CHI 2008, http://www-users.cs.umn.edu/~echi/papers/2008-CHI2008/2008-02-mech-turk-online-experiments-chi1049-kittur.pdf 
  22. ^ Noveck, Simone (2009), Wiki Government: How Technology Can Make Government Better, Democracy Stronger, and Citizens More Powerful, Brookings Institution Press 
  23. ^ Federal Transit Administration Public Transportation Participation Pilot Program, U.S. Department of Transportation, http://www.fta.dot.gov/planning/programs/planning_environment_8711.html 
  24. ^ Peer-to-Patent Community Patent Review Project, Peer-to-Patent Community Patent Review Project, http://www.peertopatent.org/ 
  25. ^ Callison-Burch, C; Dredze, M (2010), "Creating Speech and Language Data With Amazon’s Mechanical Turk", Human Language Technologies Conference: 1-12, http://www.aclweb.org/anthology-new/W/W10/W10-0701.pdf 
  26. ^ McGraw, I; Seneff, S (2011), "Growing a Spoken Language Interface on Amazon Mechanical Turk", Interspeech: 3057-3060, http://people.csail.mit.edu/jrg/2011/McGraw_Interspeech11.pdf 
  27. ^ a b c Mason, W; Suri, S (2010), "Conducting Behavioral Research on Amazon’s Mechanical Turk", Behavior Research Methods, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1691163 
  28. ^ Koblin, A (2008), "The sheep market", Creativity and Cognition, http://dl.acm.org/citation.cfm?id=1640348 
  29. ^ Explodingdog, http://www.explodingdog.com/ 
  30. ^ Linver, D. (2010), Crowdsourcing and the Evolving Relationship between Art and Artist, http://www.crowdsourcing.org/document/crowdsourcing-and-the-evolving-relationship-between-artist-and-audience/5515 
  31. ^ a b c Ross, J; Irani, L; Silberman, M.S.; Zaldivar, A; Tomlinson, B (2010). "Who are the Crowdworkers? Shifting Demographics in Mechanical Turk". CHI 2010. http://www.ics.uci.edu/~jwross/pubs/RossEtAl-WhoAreTheCrowdworkers-altCHI2010.pdf. 
  32. ^ Hirth, M; Hoßfeld, T; Train-Gia, P (2011), Human Cloud as Emerging Internet Application - Anatomy of the Microworkers Crowdsourcing Platform, http://www3.informatik.uni-wuerzburg.de/TR/tr478.pdf 
  33. ^ a b Kaufmann, N; Schulze, T; Viet, D (2011). "More than fun and money. Worker Motivation in Crowdsourcing – A Study on Mechanical Turk". Proceedings of the Seventeenth Americas Conference on Information Systems. http://schader.bwl.uni-mannheim.de/fileadmin/files/publikationen/Kaufmann_Schulze_Veit_2011_-_More_than_fun_and_money_Worker_motivation_in_Crowdsourcing_-_A_Study_on_Mechanical_Turk_AMCIS_2011.pdf. 
  34. ^ a b Ipeirotis; Provost; Wang (2010). Quality Management on Amazon Mechanical Turk. http://people.stern.nyu.edu/panos/publications/hcomp2010.pdf. 
  35. ^ Hirth; Hoßfeld; Tran-Gia (2011), Human Cloud as Emerging Internet Application - Anatomy of the Microworkers Crowdsourcing Platform, http://www3.informatik.uni-wuerzburg.de/TR/tr478.pdf 
  36. ^ "Fair Labor Standards Act Advisor". http://www.dol.gov/elaws/faq/esa/flsa/001.htm. Retrieved 28 February 2012. 
  37. ^ Norcie (2011). Ethical and Practical Considerations for Compensation of Crowdsourced Research Participants. http://www.crowdsourcing.org/document/ethical-and-practical-considerations-for-compensation-of-crowdsourced-research-participants/3650. 
  38. ^ Ipeirotis (2010). "Analyzing the Amazon Mechanical Turk Marketplace". XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter (ACM) 17 (2). http://delivery.acm.org/10.1145/1870000/1869094/p16-ipeirotis.pdf?ip=128.237.228.168&acc=OPEN&CFID=67955182&CFTOKEN=99801176&__acm__=1330412802_d8ed25958879e894447fe2c38fb1cf1a. Retrieved February 26, 2012. 
  39. ^ a b Tomoko A. Hosaka (April 2008). "Facebook asks users to translate for free". MSNBC. http://www.msnbc.msn.com/id/24205912/ns/technology_and_science-internet/. 
  40. ^ Paolacci, Chandler (2010). "Running Experiments on Amazon Mechanical Turk". Judgement and Decision Making 5 (5): 411-419. http://journal.sjdm.org/10/10630a/jdm10630a.pdf. Retrieved February 26, 2012. 
  41. ^ Lakhani et al. (2007). The Value of Openness in Scientific Problem Solving. http://www.hbs.edu/research/pdf/07-050.pdf. Retrieved February 26, 2012. 
Retrieved from "http://en.wikipedia.org/w/index.php?title=Crowdsourcing&oldid=481439888"
Personal tools
Namespaces

Variants
Actions
Navigation
Interaction
Toolbox
Print/export
Languages