Crowdsourcing

From Wikipedia, the free encyclopedia
Jump to: navigation, search

Crowdsourcing is the process of obtaining needed services, ideas, or content by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees or suppliers.[1] While this definition from Merriam Webster is valid, a more specific definition is heavily debated.[2] The process of crowdsourcing is often used to subdivide tedious work[3] and has occurred successfully offlinesee the examples below. It combines the efforts of numerous self-identified volunteers or part-time workers, where each contributor of their own initiative adds a small portion to the greater result. The term "crowdsourcing" is a portmanteau of "crowd" and "outsourcing"; it is distinguished from outsourcing in that the work comes from an undefined public rather than being commissioned from a specific, named group.

Coined in 2005, the word "crowdsourcing" can apply to a wide range of activities.[4] Crowdsourcing can involve division of labor for tedious tasks split to use crowd-based outsourcing, but it can also apply to specific requests, such as crowdfunding, a broad-based competition, and a general search for answers, solutions, or a missing person. Crowdtesting is another example of the utilization of the crowd to provide software testing services, crowdtesting is becoming a major player in the software world with recent studies stating that 55% of companies have adopted crowdsourced services in 2014 and more plan to utilize crowdtesters in 2015 and moving forward.

Definitions[edit]

Jeff Howe and Mark Robinson, editors at Wired Magazine, coined the term "crowdsourcing" in 2005 after conversations about how businesses were using the Internet to outsource work to individuals.[4] Howe and Robinson came to the conclusion that what was happening was like "outsourcing to the crowd," which quickly led to the portmanteau "crowdsourcing." Howe first published a definition for the term "crowdsourcing" in a companion blog post to his June 2006 Wired magazine article, "The Rise of Crowdsourcing," which came out in print just days later:[5]

"Simply defined, crowdsourcing represents the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. This can take the form of peer-production (when the job is performed collaboratively), but is also often undertaken by sole individuals. The crucial prerequisite is the use of the open call format and the large network of potential laborers."

In a February 1, 2008 article, Daren C. Brabham, "the first [person] to publish scholarly research using the word crowdsourcing" and writer of the 2013 book, Crowdsourcing, defined it as an "online, distributed problem-solving and production model."[6][7]

In crowdsourcing, problems are broadcast to the public in the form of an open call for solutions. Members of the public submit solutions which are then owned by the entity which broadcast the problem. In some cases, the contributor of the solution is compensated monetarily, with prizes or with recognition. In other cases, the only rewards may be kudos or intellectual satisfaction. Crowdsourcing may produce solutions from amateurs or volunteers, working in their spare time, or from experts or small businesses which were unknown to the initiating organization.[3]

Crowdsourcers are primarily motivated by its benefits. One of these includes the ability to gather large numbers of solutions and information at a relatively inexpensive cost. Users are motivated to contribute to crowdsourced tasks by both intrinsic motivations, such as social contact, intellectual stimulation, and passing time, and by extrinsic motivations, such as financial gain.

Due to the blurred limits of crowdsourcing, many collaborative activities are considered crowdsourcing even when they are not. Another consequence of this situation is the proliferation of definitions in the scientific literature.[2] Different authors give different definitions of crowdsourcing according to their specialties, losing in this way the global picture of the term.

After studying more than 40 definitions of crowdsourcing in the scientific and popular literature, Enrique Estellés-Arolas and Fernando González Ladrón-de-Guevara, researchers at the Technical University of Valencia, developed a new integrating definition:[2]

"Crowdsourcing is a type of participative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, the voluntary undertaking of a task. The undertaking of the task, of variable complexity and modularity, and in which the crowd should participate bringing their work, money, knowledge and/or experience, always entails mutual benefit. The user will receive the satisfaction of a given type of need, be it economic, social recognition, self-esteem, or the development of individual skills, while the crowdsourcer will obtain and utilize to their advantage that which the user has brought to the venture, whose form will depend on the type of activity undertaken".

Henk van Ess, a college lecturer in online communications, emphasizes the need to "give back" the crowdsourced results to the public on ethical grounds. His non-scientific, non-commercial definition is widely cited in the popular press:[8]

"Crowdsourcing is channeling the experts’ desire to solve a problem and then freely sharing the answer with everyone."

Crowdsourcing systems are used to accomplish a variety of tasks. For example, the crowd may be invited to develop a new technology, carry out a design task (also known as "community-based design"[9] or distributed participatory design), refine or carry out the steps of an algorithm (see human-based computation), or help capture, systematize, or analyze large amounts of data (see also citizen science).

Theoretical Perspectives[edit]

The Crowd Capital perspective explains how and why organizations engage Crowds through IT to generate heterogeneous reources. The Crowd Capital perspective parsimoniously generalizes the dynamics of numerous substantive research areas focused on IT-mediated Crowds including; Crowdsourcing, Citizen Science, Prediction Markets, Open Innovation platforms, Crowdfunding, and Wikis.[10][11]

Building on the work of Friedrich Hayek, the Crowd Capital perspective views dispersed knowledge as the antecedent construct representing the resources that are possible to obtain from Crowds. Crowds are engaged through an organization's Crowd Capability.[12] The Crowd Capability construct contains three dimensions; structure, content, and process, representing the components of engaging a Crowd. Crowd Capital is the outcome construct of the perspective and represents the potential resources generated when an organization engages dispersed knowledge through Crowd Capability. Crowd Capital is always an IT-mediated phenomenon, without IT, it is not possible to generate Crowd Capital. Therefore, engaging Crowds without IT, for example in-person, cannot generate Crowd Capital. See the image below for a depiction of Crowd Capital theory, adapted from A Computational Model of Crowds for Collective Intelligence.[13]

The Constructs of Crowd Capital Theory

Crowd Capability[edit]

The Crowd Capability construct contains three dimensions; structure, content, and process, representing the components of engaging a Crowd. The structure component represents the IT used to engage a Crowd, the content dimension represents the type of input that is desired from a Crowd, while the process dimension represents the organizational processes used to form resources from the Crowd.[12] All three components of Crowd Capability must be implemented by an organization to potentially generate heterogeneous resources from dispersed knowledge. A systemic perspective encompassing all elements of the Crowd Capital theory constructs is displayed below.[14]

The Constructs of Crowd Capital Theory - Systemic View

IT Structures[edit]

The Crowd Capital perspective further distinguishes that the structure dimension of Crowd Capability can be found to exist in one of two forms; Episodic or Collaborative.[15] Episodic IT structures imply that Crowd participants do not interact with one another through the IT, while Collaborative structures imply that participants in Crowds must interact with one another for resources to be generated from Crowds. Examples of Episodic structures include applications like reCAPTCHA and Foldit, whereas Enterprise wikis or the use of Social media for inputs illustrate Collaborative structures.[15]

Participation in Crowdsourcing[edit]

Despite the potential global reach of IT applications online, recent research illustrates that differences in location affect participation outcomes in IT-mediated Crowds.[14]

Policy Crowdsourcing[edit]

Research is emerging that focuses on the use of IT-mediated Crowds for policy purposes including the use of Virtual Labor Markets for policy assessment purposes, investigated experimentally.[16][17][18] In addition, the first conference focusing on the subject took place at Oxford University, under the auspices of the Oxford Internet Institute in 2014.

Historical examples[edit]

While the term "crowdsourcing" was popularized on the Internet to describe Internet-based activities,[7] there are examples of projects that in retrospect can be described as crowdsourcing.

Timeline of major events[edit]

Brief timeline of events prior to 2006

  • 1714 – The Longitude Prize: When the British government was trying to find a way to measure a ship’s longitude, they offered the public a monetary prize to whomever came up with the best solution.[19]
  • 1783 – King Louis XVI offered an award to the person who could ‘make the alkali’ by decomposing sea salt by the ‘simplest and most economic method.’[19]
  • 1884 – Publication of the Oxford English Dictionary: 800 volunteers catalogued words to create the first fascicle of the OED[19]
  • 1916 – Planters Peanuts contest: The Mr. Peanut logo was designed by a fourteen-year old boy who won the Planter Peanuts logo contest.[19]
  • 1957 – “Jom Utzon selected as winner for design competition for Sydney Opera House” [19]
  • 1996 – The Hollywood Stock Exchange was founded: Allowed for the buying and selling of shares.[19]
  • 1997 – British rock band Marillion raised $60,000 from their fans to help finance their U.S. tour.[19]
  • 2000 – JustGiving established: This online platform allows the public to help raise money for charities.[19]
  • 2000 – UNV Online Volunteering service launched: Connecting people who commit their time and skills over the Internet to help organizations address development challenges[20]
  • 2000 – iStockPhoto was founded: The free stock imagery website allows the public to contribute to and receive commission for their contributions.[21]
  • 2001 – Launch of Wikipedia: “Free-access, free content Internet encyclopedia”[22]
  • 2004 – Toyota’s first “Dream car art” contest: Children were asked globally to draw their ‘dream car of the future.’[23]
  • 2005 – Kodak’s “Go for the Gold” contest: Kodak asked anyone to submit a picture of a personal victory.[23]
  • 2006 – Jeff Howe coined the term crowdsourcing in Wired (magazine).[21]

The Oxford English Dictionary[edit]

The Oxford English Dictionary (OED) may provide one of the earliest examples of crowdsourcing. In the mid-19th century, an open call for volunteers was made for contributions identifying all words in the English language and example quotations exemplifying their usages.[24] They received over six million submissions over a period of 70 years.[25] The making of the OED is detailed in The Surgeon of Crowthorne (Published in the United States under the title "The Professor and the Madman"), by Simon Winchester.[25]

Crowdsourcing in astronomy[edit]

Crowdsourcing in Astronomy was used in the early 19th century by astronomer Denison Olmsted. After being awakened in a late November night due to a meteor shower taking place, Olmsted noticed a pattern in the shooting stars. Olmsted wrote a brief report of this meteor shower in the local newspaper. “As the cause of ‘Falling Stars’ is not understood by meteorologists, it is desirable to collect all the facts attending this phenomenon, stated with as much precision as possible,” Olmsted wrote to readers, in a report subsequently picked up and pooled to newspapers nationwide. Responses came pouring in from many states, along with scientists’ observations sent to the American Journal of Science and Arts.[26] These responses helped him make a series of scientific breakthroughs. The major being that meteor showers are seen nationwide, and fall from space under the influence of gravity. Also, they showed that the showers appeared in yearly cycles, a fact that often eluded scientists. The responses also allowed him to suggest a velocity for the meteors which turned out to be too conservative. If he had just taken the responses as presented his conjecture on the meteor's velocity would have been closer to their actual speed.

A modern day version of crowdsourcing in astronomy is NASA's photo organizing project which asks internet users to browse photos taken from space and try to identify the location the picture is documenting.[27]

Crowdsourcing in ornithology[edit]

Another early example of crowdsourcing occurred in the field of ornithology. On December 25, 1900, Frank Chapman, an early officer of the National Audubon Society initiated a tradition, dubbed the "Christmas Day Bird Census". The project called birders from across North America to count and record the number of birds in each species they witnessed on Christmas Day. The project was successful, and the records from 27 different contributors were compiled into one bird census, which tallied around 90 species of birds.[28] This large scale collection of data constituted an early form of citizen science, the premise on which crowdsourcing is based. In the 2012 census, more than 70,000 individuals participated across 2,369 bird count circles.[29] Christmas 2014 will mark the National Audubon Society's 115th annual Christmas Bird Count.

Crowdsourcing in genealogy research[edit]

Genealogical research was using crowdsourcing techniques long before personal computers were common. Beginning in 1942, members of The Church of Jesus Christ of Latter-day Saints (LDS Church) encouraged members to submit information about their ancestors. The submitted information was gathered together into a single collection. In 1969 in order to encourage more people to participate in gathering genealogical information about their ancestors, the church started the three-generation program. In this program church members were asked to prepare documented family group record forms for the first three generations. The program was later expanded to encourage members to research at least 4 generations and became known as the four-generation program.[30]

Institutes that have records of interest to genealogical research have used crowds of volunteers to create catalogs and indexes to records.

Crowdsourcing in genetic genealogy research[edit]

Genetic genealogy is a combination of traditional genealogy with Genetics. With the rise of personal DNA testing, after the turn of the century, by companies such as Gene by Gene, FTDNA, GeneTree, 23andMe, and Ancestry.com led to public and semi-public databases of DNA testing which uses crowdsourcing techniques. In recent years, citizen science projects have become increasingly focused providing benefits to scientific research.[31][32][33] This includes supporting, organization and dissemination of personal DNA (genetic) testing. Like Amateur astronomy, citizen scientists encouraged by volunteer organizations like ISOGG - the International Society of Genetic Genealogy,[34] have provided valuable information and research to the professional scientific community.[35]

Spencer Wells, Phd., Director of the Genographic Project blurb:

Since 2005, the Genographic Project has used the latest genetic technology to expand our knowledge of the human story, and its pioneering use of DNA testing to engage and involve the public in the research effort has helped to create a new breed of "citizen scientist." Geno 2.0 expands the scope for citizen science, harnessing the power of the crowd to discover new details of human population history.[36]

Early crowdsourcing competitions[edit]

Crowdsourcing has often been used in the past as a competition in order to discover a solution. The French government proposed several of these competitions, often rewarded with Montyon Prizes, created for poor Frenchmen who had done virtuous acts.[37] These included the Leblanc process, or the Alkali Prize, where a reward was provided for separating the salt from the alkali, and the Fourneyron's Turbine, when the first hydraulic commercial turbine was developed.[38]

In response to a challenge from the French government, Nicolas Appert won a prize for inventing a new way of food preservation that involved sealing food in air-tight jars.[39] The British government provided a similar reward to find an easy way to determine a ship's longitude in the The Longitude Prize. During the Great Depression, out-of-work clerks tabulated higher mathematical functions in the Mathematical Tables Project as an outreach project.[40] One of the biggest crowdsourcing campaigns was a public design contest in 2010 hosted by the Indian Government's finance ministry to create a symbol for the Indian rupee. Thousands of people sent in entries before the government zeroed in on the final symbol based on the Devanagari script using the letter Ra.[41]

Modern methods[edit]

Today, crowdsourcing has transferred mainly to the Internet. The Internet provides a particularly good venue for crowdsourcing since individuals tend to be more open in web-based projects where they are not being physically judged or scrutinized and thus can feel more comfortable sharing. This ultimately allows for well-designed artistic projects because individuals are less conscious, or maybe even less aware, of scrutiny towards their work. In an online atmosphere, more attention can be given to the specific needs of a project, rather than spending as much time in communication with other individuals.[42]

According to a definition by Henk van Ess:[43]

"The crowdsourced problem can be huge (epic tasks like finding alien life or mapping earthquake zones) or very small ('where can I skate safely?'). Some examples of successful crowdsourcing themes are problems that bug people, things that make people feel good about themselves, projects that tap into niche knowledge of proud experts, subjects that people find sympathetic or any form of injustice."

Crowdsourcing can either take an explicit or an implicit route. Explicit crowdsourcing lets users work together to evaluate, share and build different specific tasks, while implicit crowdsourcing means that users solve a problem as a side effect of something else they are doing.

With explicit crowdsourcing, users can evaluate particular items like books or webpages, or share by posting products or items. Users can also build artifacts by providing information and editing other people's work.

Implicit crowdsourcing can take two forms: standalone and piggyback. Standalone allows people to solve problems as a side effect of the task they are actually doing, whereas piggyback takes users' information from a third-party website to gather information.[44]

In his 2013 book, Crowdsourcing, Daren C. Brabham puts forth a problem-based typology of crowdsourcing approaches:[45]

  • Knowledge Discovery & Management - for information management problems where an organization mobilizes a crowd to find and assemble information. Ideal for creating collective resources.
  • Distributed Human Intelligence Tasking - for information management problems where an organization has a set of information in hand and mobilizes a crowd to process or analyze the information. Ideal for processing large data sets that computers cannot easily do.
  • Broadcast Search - for ideation problems where an organization mobilizes a crowd to come up with a solution to a problem that has an objective, provable right answer. Ideal for scientific problem solving.
  • Peer-Vetted Creative Production - for ideation problems where an organization mobilizes a crowd to come up with a solution to a problem which has an answer that is subjective or dependent on public support. Ideal for design, aesthetic, or policy problems.

Examples of crowdsourcing[edit]

There are some common categories of crowdsourcing that can be used effectively in the commercial world. Some of these web-based crowdsourcing efforts include crowdvoting, crowdfunding, microwork, creative crowdsourcing, Crowdsource Workforce Management and inducement prize contests. Although these may not be an exhaustive list, they cover the current major ways in which people use crowds to perform tasks.[46]

Crowdvoting[edit]

Crowdvoting occurs when a website gathers a large group's opinions and judgment on a certain topic. The Iowa Electronic Market is a prediction market that gathers crowds' views on politics and tries to ensure accuracy by having participants pay money to buy and sell contracts based on political outcomes.[47]

Some of the most famous examples have made use of social media channels: Domino's Pizza, Coca Cola, Heineken and Sam Adams have thus crowdsourced a new pizza, bottle design, beer and song, respectively.[48] Threadless.com selects the T-shirts it sells by having users provide designs and vote on the ones they like, which are then printed and available for purchase.[7]

The California Report Card (CRC), a program jointly launched in January 2014 by the Center for Information Technology Research in the Interest of Society[49] and Lt. Governor Gavin Newsom, is an example of modern-day crowd voting. Participants access the CRC on line and vote on six timely issues. Through principal component analysis, the users are then placed into an online "café" in which they can present their own political opinions and grade the suggestions of other participants. This system aims to effectively involve the greater public in relevant political discussions and highlight the specific topics with which Californians are most concerned.

Crowdsourcing creative work[edit]

Creative crowdsourcing spans sourcing creative projects such as graphic design, crowdsourcing architecture, apparel design, writing, illustration, etc.[50][51]

Crowdsourcing language-related data collection[edit]

Crowdsourcing has also been used for gathering language-related data. For dictionary work, as was mentioned above, over a hundred years ago it was applied by the Oxford English Dictionary editors, using paper and postage. Much later, a call for collecting examples of proverbs on a specific topic (religious pluralism) was printed in a journal.[52] Today, as "crowdsourcing" has the inherent connotation of being Web-based, such language-related data gathering is being conducted on the Web by crowdsourcing in accelerating ways. Currently, there are a number of dictionary compilation projects being conducted on the Web, particularly for languages that are not highly academically documented, such as for the Oromo language.[53] Software programs have been developed for crowdsourced dictionaries, such as WeSay.[54] A slightly different form of crowdsourcing for language data has been the online creation of scientific and mathematical terminology for American Sign Language.[55] Proverb collection is also being done via crowdsourcing on the Web, most innovatively for the Pashto language of Afghanistan and Pakistan.[56] Crowdsourcing has been extensively used to collect high-quality gold standard for creating automatic systems in natural language processing (e.g., named entity recognition, entity linking).[57]

Crowdsearching[edit]

Chicago-based startup crowdfynd utilizes a version of crowdsourcing best termed as crowdsearching, which differs from Microwork in that there is no obligated payment for taking part in the search.[58] Their platform, through geographic location anchoring, builds a virtual search party of smartphone and internet users to find a lost item, pet or person, as well as returning a found item, pet or property.

Crowdfunding[edit]

Crowdfunding is the process of funding your projects by a multitude of people contributing a small amount in order to attain a certain monetary goal, typically via the Internet.[59] Two basic crowdfunding models exist. The model that has been around the longest is rewards-based crowdfunding. This is where people can pre-purchase products, buy experiences, or simply donate. While this funding may in some cases go towards helping a business, funders are not allowed to invest and become shareholders via rewards-based crowdfunding.[60]

Individuals, businesses, and entrepreneurs can showcase their businesses and projects to the entire world by creating a profile, typically includes a short video, introducing their project, a list of rewards per donation, and illustrations through images. The idea is to create a compelling message that readers will be drawn towards. Funders make monetary contribution for numerous reasons:

1. The connect to the greater purpose of the campaign

2. They connect to a physical aspect of the campaign like rewards

3. They connect to the creative display of the campaign’s presentation


The dilemma for equity crowdfunding in the US as of 2012 was how the SEC is going to regulate the entire process. At the time, rules and regulations were being refined by the SEC, which had until Jan. 1st, 2013 to tweak the fundraising methods. The regulators were overwhelmed trying to regulate Dodd – Frank and all the other rules and regulations involving public companies and the way they trade. Advocates of regulation claimed that crowdfunding would open up the flood gates for fraud, called it the "wild west" of fundraising, and compared it to the 1980s days of penny stock "cold-call cowboys." The process allows for up to 1 million dollars to be raised without a lot of the regulations being involved. Companies under the then-current proposal would have a lot of exemptions available and be able to raise capital from a larger pool of persons, which can include a lot lower thresholds for investor criteria whereas the old rules required that the person be an "accredited" investor. These people are often recruited from social networks, where the funds can be acquired from an equity purchase, loan, donation, or pre-ordering. The amounts collected have become quite high, with requests that are over a million dollars for software like Trampoline Systems, which used it to finance the commercialization of their new software.[54]

Crowdfunding sites [60]

Kickstarter is a new way to fund creative projects. We’re a home for films, games, and music to art, design, and technology.

Indiegogo is open to almost any kind of project (they even crowdfunded a baby [61]), and the company has a larger international presence than Kickstarter

Crowdrise is a platform for donating to charitable causes.

Quirky is a rewards-based crowdfunding platform and online community most often used by product inventors and makers.

Tilt is the rewards-based crowdfunding solution for groups and communities to pool their dollars and raise money together.

Invested.in is the place to go if you’d like to create your own crowdfunding platform by “white-labeling” their software.

Macrowork[edit]

Macrowork tasks typically have the following characteristics: they can be done independently; they take a fixed amount of time; and they require special skills. Macrotasks could be part of specialized projects or could be part of a large, visible project where workers pitch in wherever they have the required skills. The key distinguishing factors are that macrowork requires specialized skills and typically takes longer, while microwork requires no specialized skills.

Microwork[edit]

Microwork is a crowdsourcing platform where users do small tasks for which computers lack aptitude for low amounts of money. Amazon’s popular Mechanical Turk has created many different projects for users to participate in, where each task requires very little time and offers a very small amount in payment.[3] The Chinese versions of this, commonly called Witkey, are similar and include such sites as Taskcn.com and k68.cn. When choosing tasks, since only certain users “win”, users learn to submit later and pick less popular tasks in order to increase the likelihood of getting their work chosen.[62] An example of a Mechanical Turk project is when users searched satellite images for a boat in order to find lost researcher Jim Gray.[44]

Inducement prize contests[edit]

Web-based idea competitions or inducement prize contests often consist of generic ideas, cash prizes, and an Internet-based platform to facilitate easy idea generation and discussion. An example of these competitions includes an event like IBM's 2006 "Innovation Jam", attended by over 140,000 international participants and yielding around 46,000 ideas.[63][64] Another example is the Netflix Prize in 2009. The idea was to ask the crowd to come up with a recommendation algorithm as more accurate than Netflix's own algorithm. It had a grand prize of US$1,000,000, and it was given to the BellKor's Pragmatic Chaos team which bested Netflix's own algorithm for predicting ratings, by 10.06%.

Another example of competition-based crowdsourcing is the 2009 DARPA balloon experiment, where DARPA placed 10 balloon markers across the United States and challenged teams to compete to be the first to report the location of all the balloons. A collaboration of efforts was required to complete the challenge quickly and in addition to the competitive motivation of the contest as a whole, the winning team (MIT, in less than nine hours) established its own "collaborapetitive" environment to generate participation in their team.[65] A similar challenge was the Tag Challenge, funded by the US State Department, which required locating and photographing individuals in 5 cities in the US and Europe within 12 hours based only on a single photograph. The winning team managed to locate 3 suspects by mobilizing volunteers world-wide using a similar incentive scheme to the one used in the Balloon Challenge.[66]

Open innovation platforms are a very effective way of crowdsourcing people's thoughts and ideas to do research and development. The company InnoCentive is a crowdsourcing platform for corporate research and development where difficult scientific problems are posted for crowds of solvers to discover the answer and win a cash prize, which can range from $10,000 to $100,000 per challenge.[7] InnoCentive, of Waltham, MA and London, England is the leader in providing access to millions of scientific and technical experts from around the world. The company has provided expert crowdsourcing to international Fortune 1000 companies in the US and Europe as well as government agencies and nonprofits. The company claims a success rate of 50% in providing successful solutions to previously unsolved scientific and technical problems. IdeaConnection.com challenges people to come up with new inventions and innovations and Ninesigma.com connects clients with experts in various fields. The X PRIZE Foundation creates and runs incentive competitions where one can win between $1 million and $30 million for solving challenges. Local Motors is another example of crowdsourcing. A community of 20,000 automotive engineers, designers and enthusiasts competes to build offroad rally trucks.[67]

Implicit crowdsourcing[edit]

Implicit crowdsourcing is less obvious because users do not necessarily know they are contributing, yet can still be very effective in completing certain tasks. Rather than users actively participating in solving a problem or providing information, implicit crowdsourcing involves users doing another task entirely where a third party gains information for another topic based on the user's actions.[7]

A good example of implicit crowdsourcing is the ESP game, where users guess what images are and then these labels are used to tag Google images. Another popular use of implicit crowdsourcing is through reCAPTCHA, which asks people to solve CAPTCHAs to prove they are human, and then provides CAPTCHAs from old books that cannot be deciphered by computers, to digitize them for the web. Like many tasks solved using the Mechanical Turk, CAPTCHAs are simple for humans but often very difficult for computers.[44]

Piggyback crowdsourcing can be seen most frequently by websites such as Google that data-mine a user's search history and websites in order to discover keywords for ads, spelling corrections, and finding synonyms. In this way, users are unintentionally helping to modify existing systems, such as Google's AdWords.[68]

Crowdsourcers[edit]

There are a number of motivations for businesses to use crowdsourcing to accomplish their tasks, find solutions for problems, or to gather information. These include the ability to offload peak demand, access cheap labor and information, generate better results, access a wider array of talent than might be present in one organization, and undertake problems that would have been too difficult to solve internally.[69] Crowdsourcing allows businesses to submit problems on which contributors can work, on topics such as science, manufacturing, biotech, and medicine, with monetary rewards for successful solutions. Although it can be difficult to crowdsource complicated tasks, simple work tasks can be crowdsourced cheaply and effectively.[70]

Crowdsourcing also has the potential to be a problem-solving mechanism for government and nonprofit use.[71] Urban and transit planning are prime areas for crowdsourcing. One project to test crowdsourcing's public participation process for transit planning in Salt Lake City was carried out from 2008 to 2009, funded by a U.S. Federal Transit Administration grant.[72] Another notable application of crowdsourcing to government problem solving is the Peer to Patent Community Patent Review project for the U.S. Patent and Trademark Office.[73]

Researchers have used crowdsourcing systems (in particular, the Mechanical Turk) to aid with research projects by crowdsourcing some aspects of the research process, such as data collection, parsing, and evaluation. Notable examples include using the crowd to create speech and language databases,[74][75] and using the crowd to conduct user studies.[68] Crowdsourcing systems provide these researchers with the ability to gather large amount of data. Additionally, using crowdsourcing, researchers can collect data from populations and demographics they may not have had access to locally, but that improve the validity and value of their work.[76]

Artists have also utilized crowdsourcing systems. In his project the Sheep Market, Aaron Koblin used Mechanical Turk to collect 10,000 drawings of sheep from contributors around the world.[77] Sam Brown (artist) leverages the crowd by asking visitors of his website explodingdog to send him sentences that he uses as inspirations for paintings.[78] Art curator Andrea Grover argues that individuals tend to be more open in crowdsourced projects because they are not being physically judged or scrutinized.[42] As with other crowdsourcers, artists use crowdsourcing systems to generate and collect data. The crowd also can be used to provide inspiration and to collect financial support for an artist's work.[79]

Additionally, crowdsourcing from 100 million drivers is being used by INRIX to collect users' driving times to provide better GPS routing and real-time traffic updates.[80]

Demographics[edit]

The crowd is an umbrella term for the people who contribute to crowdsourcing efforts. Though it is sometimes difficult to gather data about the demographics of the crowd, a study by Ross et al. surveyed the demographics of a sample of the more than 400,000 registered crowdworkers using Amazon Mechanical Turk to complete tasks for pay. While a previous study in 2008 by Ipeirotis found that users at that time were primarily American, young, female, and well-educated, with 40% having incomes >$40,000/yr, in 2009 Ross found a very different population. By Nov. 2009, 36% of the surveyed Mechanical Turk workforce was Indian. Two-thirds of Indian workers were male, and 66% had at least a Bachelor’s degree. Two-thirds had annual incomes less than $10,000/yr, with 27% sometimes or always depending on income from Mechanical Turk to make ends meet.[81]

The average US user of Mechanical Turk earned $2.30 per hour for tasks in 2009, versus $1.58 for the average Indian worker.[citation needed] While the majority of users worked less than five hours per week, 18% worked 15 hours per week or more. This is less than minimum wage in either country, which Ross suggests raises ethical questions for researchers who use crowdsourcing.

The demographics of http://microworkers.com/ differ from Mechanical Turk in that the US and India together account for only 25% of workers. 197 countries are represented among users, with Indonesia (18%) and Bangladesh (17%) contributing the largest share. However, 28% of employers are from the US.[82]

Another study of the demographics of the crowd at iStockphoto found a crowd that was largely white, middle- to upper-class, higher educated, worked in a so-called "white collar job" and had a high-speed Internet connection at home.[83]

Studies have also found that crowds are not simply collections of amateurs or hobbyists. Rather, crowds are often professionally trained in a discipline relevant to a given crowdsourcing task and sometimes hold advanced degrees and many years of experience in the profession.[83][84][85][86] Claiming that crowds are amateurs, rather than professionals, is both factually untrue and may lead to marginalization of crowd labor rights.[87]

G. D. Saxton et al. (2013) studied the role of community users, among other elements, during his content analysis of 103 crowdsourcing organizations. Saxton et al. developed a taxonomy of nine crowdsourcing models (intermediary model, citizen media production, collaborative software development, digital goods sales, product design, peer-to-peer social financing, consumer report model, knowledge base building model, and collaborative science project model) in which to categorize the roles of community users, such as researcher, engineer, programmer, journalist, graphic designer, etc., and the products and services developed.[88]

Motivations[edit]

Many scholars of crowdsourcing suggest that there are both intrinsic and extrinsic motivations that cause people to contribute to crowdsourced tasks and that these factors influence different types of contributors.[83][84][86][89][90][91] For example, students and people employed full-time rate Human Capital Advancement as less important than part-time workers do, while women rate Social Contact as more important than men do.[89]

Intrinsic motivations are broken down into two categories: enjoyment-based and community-based motivations. Enjoyment-based motivations refer to motivations related to the fun and enjoyment that the contributor experiences through their participation. These motivations include: skill variety, task identity, task autonomy, direct feedback from the job, and pastime. Community-based motivations refer to motivations related to community participation, and include community identification and social contact.

Extrinsic motivations are broken down into three categories: immediate payoffs, delayed payoffs, and social motivations. Immediate payoffs, through monetary payment, are the immediately received compensations given to those who complete tasks. Delayed payoffs are benefits that can be used to generate future advantages, such as training skills and being noticed by potential employers. Social motivations are the rewards of behaving pro-socially,[92] such as the altruistic motivations of online volunteers. Chandler and Kapelner found that US users of the Amazon Mechanical Turk were more likely to complete a task when told they were going to “help researchers identify tumor cells,” than when they were not told the purpose of their task. However, of those who completed the task, quality of output did not depend on the framing of the task.[89]

Another form of social motivation is prestige or status. The International Children's Digital Library recruits volunteers to translate and review books. Because all translators receive public acknowledgment for their contributions, Kaufman and Schulz cite this as a reputation-based strategy to motivate individuals who want to be associated with institutions that have prestige. The Amazon Mechanical Turk uses reputation as a motivator in a different sense, as a form of quality control. Crowdworkers who frequently complete tasks in ways judged to be inadequate can be denied access to future tasks, providing motivation to produce high-quality work.[93]

Controversies[edit]

There are four major topics of controversy about crowdsourcing: (1)entrepreneurs contribute less capital themselves, (2)increased number of funded ideas, (3) the value and impact of the work received from the crowd, and (4) the ethical implications of low wages paid to crowdworkers.

Entrepreneurs Contribute Less Capital Themselves[edit]

To make an idea turn into a reality, the first thing needed is capital. Depending on the scope and complexity of the crowdsourced project, the amount of necessary capital can range from a few thousand dollars to hundreds of thousands. The capital-raising process can take anywhere from days to months depending on different variables including the entrepreneur’s network and the amount of initial self-generated capital. The crowdsourcing process allows entrepreneurs to access to a wide range of investors who can take different stakes in the project.[94] In effect, crowdsourcing simplifies the capital raising process and allows entrepreneurs to spend more time on the project itself and reaching milestones rather than dedicating time to get it started. Overall, the simplified access to capital can save time to start projects and potentially increase efficiency of projects. Opponents of this issue argue easier access to capital through a large number of smaller investors can hurt the project and its creators. With a simplified capital raising process involving more investors with smaller stakes, investors are more risk seeking due to the fact that they can take on an investment size they are comfortable with.[95] This leads to entrepreneurs losing possible experience convincing investors who are wary of potential risks in investing because they do not depend on one single investor for the survival of their project. Instead of being forced to assess risks and convince large institutional investors why their project can be successful, wary investors can be replaced by others who are willing to take on the risk.

Increased Number of Funded Ideas[edit]

The raw number of ideas that get funded and the quality of the ideas is a large controversy over the issue of crowdsourcing. Proponents argue crowdsourcing is beneficial because it allows niche ideas that would not survive Venture Capitalist or Angel funding, many times the primary investors in startups, to be started. Many ideas are killed in their infancy due to insufficient support and lack of capital but crowdsourcing allows these ideas to be started if a entrepreneur can find a community to take interest in the project.[96] Crowdsourcing allows those who would benefit from the project to fund and become a part of it which is one way how small niche ideas get started.[97] On the other hand, when the raw number of projects grow, the number of possible failures can also increase. Crowdsourcing assists niche and high-risk projects to start because of a perceived need from a select few who seek the product. With high risk and small target markets, the pool of crowdsourced projects face a greater possible loss of capital, lower return, and lower levels of success.[98]

Impact of crowdsourcing on product quality[edit]

There is susceptibility to faulty results caused by targeted, malicious work efforts. Since crowdworkers completing microtasks are paid per task, there is often a financial incentive to complete tasks quickly rather than well. Verifying responses is time-consuming, and so requesters often depend on having multiple workers complete the same task to correct errors. However, having each task completed multiple times increases time and monetary costs.[99] Along with adding worker redundancy, it has also been showed that estimating crowdworkers' skills and intentions and leveraging them for inferring true responses works well,[100] albeit with an additional computation cost.

Crowdworkers are a nonrandom sample of the population. Many researchers use crowdsourcing in order to quickly and cheaply conduct studies with larger sample sizes than would be otherwise achievable. However, due to limited access to internet, participation in low developed countries is relatively low. Participation in highly developed countries is similary low, largely because the low amount of pay is not a strong motivation for most users in these countries. These factors lead to a bias in the population pool towards users in medium developed countries, as deemed by the Human Development Index.[101]

Increased likelihood that a crowdsourced project will fail due to lack of monetary motivation or too few participants. Crowdsourcing markets are not a first-in-first-out queue. Tasks that are not completed quickly may be forgotten, buried by filters and search procedures so that workers do not see them. This results in a long tail power law distribution of completion times.[102] Additionally, low-paying research studies online have higher rates of attrition, with participants not completing the study once started.[76] Even when tasks are completed, crowdsourcing does not always produce quality results. When Facebook began its localization program in 2008, it encountered some criticism for the low quality of its crowdsourced translations.[103]

One of the problems of crowdsourcing products is the lack of interaction between the crowd and the client. Usually there is little information about the final desired product, and there is often very limited interaction with the final client. This can decrease the quality of product because client interaction is a vital part of the design process.[104]

It is usually expected from a crowdsourced project to be unbiased by incorporating a large population of participants with a diverse background. However, most of the crowdsourcing works are done by people who are paid or directly benefit from the outcome (e.g. most of open source projects working on Linux). In many other cases, the end product is the outcome of a single person's endeavour, who creates the majority of the product, while the crowd only participates in minor details.[105]

Ethical concerns for crowdsourcers[edit]

Because crowdworkers are considered independent contractors rather than employees, they are not guaranteed minimum wage. In practice, workers using the Amazon Mechanical Turk generally earn less than the minimum wage, with US users earning an average of $2.30 per hour for tasks in 2009, and users in India earning an average of $1.58 per hour, which is below minimum wage in both countries.[81][106] Some researchers who have considered using Mechanical Turk to get participants for research studies have argued that the wage conditions might be unethical.[76][107] However, according to other research, workers on Amazon Mechanical Turk don't feel that they are exploited and ready to participate in Crowdsourcing activities in the future.[108] When Facebook began its localization program in 2008, it received criticism for using free labor in crowdsourcing the translation of site guidelines.[103]

Typically, no written contracts, non-disclosure agreements, or employee agreements are made with crowdworkers. For users of the Amazon Mechanical Turk, this means that requestors decide whether users' work is acceptable, and reserve the right to withhold pay if it does not meet their standards.[109] Critics claim that crowdsourcing arrangements exploit individuals in the crowd, and there has been a call for crowds to organize for their labor rights.[87]

Collaboration between crowd members can also be difficult or even discouraged, especially in the context of competitive crowd sourcing. Crowdsourcing site InnoCentive allows organizations to solicit solutions to scientific and technological problems; only 10.6% of respondents report working in a team on their submission.[84]

See also[edit]

References[edit]

  1. ^ "Crowdsourcing - Definition and More". Merriam-Webster.com. August 31, 2012. Retrieved 2014-02-03. 
  2. ^ a b c Estellés-Arolas, Enrique; González-Ladrón-de-Guevara, Fernando (2012), Towards an Integrated Crowdsourcing Definition (PDF), Journal of Information Science 38 (2): 189–200, doi:10.1177/0165551512437638 
  3. ^ a b c Howe, Jeff (2006). "The Rise of Crowdsourcing". Wired. 
  4. ^ a b Safire, William (February 5, 2009). "On Language". New York Times Magazine. Retrieved May 19, 2013. 
  5. ^ Howe, Jeff (June 2, 2006). "Crowdsourcing: A Definition". Crowdsourcing Blog. Retrieved January 2, 2013. 
  6. ^ "Daren C. Brabham". USC Annenberg. University of Southern California. Retrieved 17 September 2014. 
  7. ^ a b c d e Brabham, Daren (2008), Crowdsourcing as a Model for Problem Solving: An Introduction and Cases (PDF), Convergence: The International Journal of Research into New Media Technologies 14 (1): 75–90, doi:10.1177/1354856507084420 
  8. ^ Claypole, Maurice (February 14, 2012). "Learning through crowdsourcing is deaf to the language challenge". The Guardian (London). 
  9. ^ Whitford, David (January 8, 2010). "Crowd Sourcing Turns Business On Its Head". CNN. Retrieved February 27, 2012. 
  10. ^ Prpić, J., Jackson, P., & Nguyen, T. 2014. "A Computational Model of Crowds for Collective Intelligence.". Collective Intelligence 2014. MIT Center for Collective Intelligence. Retrieved November 8, 2014. 
  11. ^ Prpić, J., & Shukla, P. (2013). "The Theory of Crowd Capital.". Proceedings of the Hawaii International Conference on Systems Sciences #46. January 2013, Maui, Hawaii, USA. IEEE Computer Society Press. Retrieved November 8, 2014. 
  12. ^ a b Prpić, J & Shukla, P. 2014. The Contours of Crowd Capability, Proceedings of the 47th Annual Hawaii International Conference on System Sciences, Big Island Hawaii, January 6–9, Computer Society Press, 2014. Best Paper Nomination, |url=http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2324637 |accessdate= November 8, 2014
  13. ^ Prpić, J., Jackson, P., & Nguyen, T. 2014. A Computational Model of Crowds for Collective Intelligence. Collective Intelligence 2014. MIT Center for Collective Intelligence |url=http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2398206 |accessdate= November 8, 2014
  14. ^ a b Prpić, J, Shukla, P., Roth, Y., and Lemoine, J.F. (2015). A Geography of Participation in IT-Mediated Crowds. Proceedings of the Hawaii International Conference on Systems Sciences 2015.|url=http://ssrn.com/abstract=2494537%7Caccessdate= November 8, 2014
  15. ^ a b Prpić, J, Shukla, P., Kietzmann, J., and McCarthy, I.P., 2015. How to Work a Crowd: Developing Crowd Capital Through Crowdsourcing. Business Horizons. |url=http://ssrn.com/abstract=2459881 & http://www.sciencedirect.com/science/article/pii/S0007681314001438 |accessdate= November 8, 2014
  16. ^ Prpić, J., Taeihagh, A., & Melton, J. 2014. Crowdsourcing the Policy Cycle. Collective Intelligence 2014, MIT Center for Collective Intelligence |url=http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2398191%7Caccessdate= November 8, 2014
  17. ^ Prpić, J., Taeihagh, A., & Melton, J. 2014. A Framework for Policy Crowdsourcing. Oxford Internet Institute, University of Oxford - IPP 2014 - Crowdsourcing for Politics and Policy |url=http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2398191%7Caccessdate= November 8, 2014
  18. ^ Prpić, J., Taeihagh, A., & Melton, J. 2014. Experiments on Crowdsourcing Policy Assessment. Oxford Internet Institute, University of Oxford - IPP 2014 - Crowdsourcing for Politics and Policy |url=http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2433391%7Caccessdate= November 8, 2014
  19. ^ a b c d e f g h Dawson Ross, Getting Results From Crowds (2012), http://www.crowdsourcing.org/editorial/a-brief-history-of-crowdsourcing-infographic/12532
  20. ^ History of the Online Volunteering service, https://www.onlinevolunteering.org/en/org/about/history.html
  21. ^ a b Howe Jeff, The Rise of Crowdsourcing (2006), http://archive.wired.com/wired/archive/14.06/crowds.html
  22. ^ ”Wikipedia,” http://en.wikipedia.org/wiki/Wikipedia
  23. ^ a b ”Crowdsourcing Back Up Timeline,” http://www.tiki-toki.com/timeline/entry/323158/Crowdsourcing-Back-Up-Timeline-Early-Stories/
  24. ^ Winchester, Simon (1999). The Professor and the Madman. New York: HarperPerennial. ISBN 978-0060839789. 
  25. ^ a b Lanxon, Nate (January 31, 2011). "How the Oxford English Dictionary started out like Wikipedia". Retrieved 2012-04-04. 
  26. ^ Vergano, Dan. "1833 Meteor Storm Started Citizen Science". National Geographic. StarStruck. Retrieved 18 September 2014. 
  27. ^ McLaughlin, Elliot. "Image Overload: Help us sort it all out, NASA requests". Cnn.com. CNN. Retrieved 18 September 2014. 
  28. ^ [1]
  29. ^ [2]
  30. ^ "What Is the Four-Generation Program?". The Church of Jesus Christ of Latter-day Saints. Retrieved January 30, 2012. 
  31. ^ Bonney, R. and LaBranche, M. (2004). Citizen Science: Involving the Public in Research. ASTC Dimensions. May/June 2004, p. 13.
  32. ^ Baretto, C., Fastovsky, D. and Sheehan, P. (2003). A Model for Integrating the Public into Scientific Research. Journal of Geoscience Education. 50 (1). pp. 71–75.
  33. ^ McCaffrey, R.E. (2005). Using Citizen Science in Urban Bird Studies. Urban Habitats. 3 (1). pp. 70–86.
  34. ^ King, Turi E.; Jobling, Mark A. (2009). "What's in a name? Y chromosomes, surnames and the genetic genealogy revolution". Trends in Genetics 25 (8): 351–60. doi:10.1016/j.tig.2009.06.003. PMID 19665817. "The International Society of Genetic Genealogy (www.isogg.org) advocates the use of genetics as a tool for genealogical research, and provides a support network for genetic genealogists. It hosts the ISOGG Y-haplogroup tree, which has the virtue of being regularly updated." 
  35. ^ Mendex, etc. al., Fernando (28 February 2013). "An African American Paternal Lineage Adds an Extremely Ancient Root to the Human Y Chromosome Phylogenetic Tree". The American Society of Human Genetics. Retrieved 10 July 2013.  This is Volume 92, Issue 3, pp. 454–459.
  36. ^ Wells, Spencer (2013). "The Genographic Project and the Rise of Citizen Science". Southern California Genealogical Society (SCGS). Retrieved July 10, 2013. 
  37. ^ "Antoine-Jean-Baptiste-Robert Auget, Baron de Montyon". New Advent. Retrieved February 25, 2012. 
  38. ^ "It Was All About Alkali". Chemistry Chronicles. Retrieved February 25, 2012. 
  39. ^ "Nicolas Appert". John Blamire. Retrieved February 25, 2012. 
  40. ^ "9 Examples of Crowdsourcing, Before ‘Crowdsourcing’ Existed". MemeBurn. Retrieved February 25, 2012. 
  41. ^ Pande, Shamni. "The People Know Best". Business Today. India: Living Media India Limited. 
  42. ^ a b DeVun, Leah (November 19, 2009). "Looking at how crowds produce and present art.". Wired News. Retrieved February 26, 2012. 
  43. ^ Ess, Henk van "Crowdsourcing: how to find a crowd", ARD ZDF Akademie 2010, Berlin, p. 99,
  44. ^ a b c Doan, A.; Ramarkrishnan, R.; Halevy, A. (2011), Crowdsourcing Systems on the World Wide Web (PDF), Communications of the ACM 54 (4): 86–96, doi:10.1145/1924421.1924442 
  45. ^ Brabham, Daren C. (2013), Crowdsourcing, MIT Press. 
  46. ^ Howe, Jeff (2008), Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business (PDF), The International Achievement Institute. 
  47. ^ Robson, John (February 24, 2012). "IEM Demonstrates the Political Wisdom of Crowds". Canoe.ca. Retrieved March 31, 2012. 
  48. ^ "4 Great Examples of Crowdsourcing through Social Media". digitalagencymarketing.com. 2012. 
  49. ^ Goldberg, Ken; Newsom, Gavin. "Let's amplify California's collective intelligence". Citris-uc.org. Retrieved 14 June 2014. 
  50. ^ "Compete To Create Your Dream Home". FastCoexist.com. June 4, 2013. Retrieved 2014-02-03. 
  51. ^ "Designers, clients forge ties on web". Boston Herald. June 11, 2012. Retrieved 2014-02-03. 
  52. ^ Stan Nussbaum. 2003. Proverbial perspectives on pluralism. Connections: the journal of the WEA Missions Committee October, pp. 30, 31.
  53. ^ "Oromo dictionary project". OromoDictionary.com. Retrieved 2014-02-03. 
  54. ^ "Description of WeSay software and process" (PDF). Retrieved 2014-02-03. 
  55. ^ "Developing ASL vocabulary for science and math". Washington.edu. December 7, 2012. Retrieved 2014-02-03. 
  56. ^ "Pashto Proverb Collection project". AfghanProverbs.com. Retrieved 2014-02-03. 
  57. ^ "Web 2.0-based crowdsourcing for high-quality gold standard development in clinical Natural Language Processing". Jmir.org. doi:10.2196/jmir.2426. Retrieved 2014-02-03. 
  58. ^ Lombard, Amy (May 5, 2013). "Crowdfynd: The First Place to Look". TIME.com. Retrieved 2014-02-03. 
  59. ^ http://www.forbes.com/sites/tanyaprive/2012/11/27/what-is-crowdfunding-and-how-does-it-benefit-the-economy/
  60. ^ a b http://www.forbes.com/sites/chancebarnett/2014/08/29/crowdfunding-sites-in-2014/
  61. ^ https://www.indiegogo.com/projects/help-the-haleys-have-a-baby
  62. ^ Yang, J.; Adamic, L.; Ackerman, M. (2008), Crowdsourcing and Knowledge Sharing: Strategic User Behavior on Taskcn (PDF), Proceedings of the 9th ACM Conference on Electronic Commerce 
  63. ^ Leimeister, J.M.; Huber, M.; Bretschneider, U.; Krcmar, H. (2009), Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition, Journal of Management Information Systems 26 (1): 197–224, doi:10.2753/mis0742-1222260108 
  64. ^ Ebner, W.; Leimeister, J.; Krcmar, H. (2009), Community Engineering for Innovations: The Ideas Competition as a method to nurture a Virtual Community for Innovations, R&D Management 39 (4): 342–356, doi:10.1111/j.1467-9310.2009.00564.x 
  65. ^ "DARPA Network Challenge". DARPA Network Challenge. Retrieved November 28, 2011. 
  66. ^ "Social media web snares 'criminals'". New Scientist. Retrieved April 4, 2012. 
  67. ^ "Beyond XPrize: The 10 Best Crowdsourcing Tools and Technologies". February 20, 2012. Retrieved March 30, 2012. 
  68. ^ a b Kittur, A.; Chi, E.H.; Sun, B. (2008), Crowdsourcing user studies with Mechanical Turk (PDF), CHI 2008 
  69. ^ Noveck, Beth Simone (2009), Wiki Government: How Technology Can Make Government Better, Democracy Stronger, and Citizens More Powerful, Brookings Institution Press 
  70. ^ Sarasua, Cristina; Simperl, Elena; Noy, Natalya F. (2012), Crowdsourcing Ontology Alignment with Microtasks (PDF), Institute AIFB. Karlsruhe Institute of Technology: 2 
  71. ^ "Crowdfunding and Civic Society in Europe: A Profitable Partnership?". Open Citizenship Journal. Retrieved April 29, 2013. 
  72. ^ Federal Transit Administration Public Transportation Participation Pilot Program, U.S. Department of Transportation 
  73. ^ Peer-to-Patent Community Patent Review Project, Peer-to-Patent Community Patent Review Project 
  74. ^ Callison-Burch, C.; Dredze, M. (2010), Creating Speech and Language Data With Amazon’s Mechanical Turk (PDF), Human Language Technologies Conference: 1–12 
  75. ^ McGraw, I.; Seneff, S. (2011), Growing a Spoken Language Interface on Amazon Mechanical Turk (PDF), Interspeech: 3057–3060 
  76. ^ a b c Mason, W.; Suri, S. (2010), Conducting Behavioral Research on Amazon’s Mechanical Turk, Behavior Research Methods 
  77. ^ Koblin, A. (2008), The sheep market, Creativity and Cognition 
  78. ^ Explodingdog 
  79. ^ Linver, D. (2010), Crowdsourcing and the Evolving Relationship between Art and Artist 
  80. ^ http://www.inrix.com/companyoverview.asp
  81. ^ a b Ross, J.; Irani, L.; Silberman, M.S.; Zaldivar, A.; Tomlinson, B. (2010). "Who are the Crowdworkers? Shifting Demographics in Mechanical Turk" (PDF). CHI 2010. 
  82. ^ Hirth, M.; Hoßfeld, T.; Train-Gia, P. (2011), Human Cloud as Emerging Internet Application – Anatomy of the Microworkers Crowdsourcing Platform (PDF) 
  83. ^ a b c Brabham, Daren C. (2008). "Moving the Crowd at iStockphoto: The Composition of the Crowd and Motivations for Participation in a Crowdsourcing Application". First Monday. 
  84. ^ a b c Lakhani et al. (2007). "The Value of Openness in Scientific Problem Solving". Retrieved February 26, 2012. 
  85. ^ Brabham, Daren C. (2012). "Managing Unexpected Publics Online: The Challenge of Targeting Specific Groups with the Wide-Reaching Tool of the Internet". International Journal of Communication. 
  86. ^ a b Brabham, Daren C. (2010). "Moving the Crowd at Threadless: Motivations for Participation in a Crowdsourcing Application". Information, Communication & Society. doi:10.1080/13691181003624090. 
  87. ^ a b Brabham, Daren C. (2012). "The Myth of Amateur Crowds: A Critical Discourse Analysis of Crowdsourcing Coverage". Information, Communication & Society. doi:10.1080/1369118X.2011.641991. 
  88. ^ Saxton, Oh, & Kishore (2013). "Rules of Crowdsourcing: Models, Issues, and Systems of Control. Information Systems Management, 30, 2–20". doi:10.1080/10580530.2013.739883. 
  89. ^ a b c Kaufmann, N.; Schulze, T.; Viet, D. (2011). "More than fun and money. Worker Motivation in Crowdsourcing – A Study on Mechanical Turk" (PDF). Proceedings of the Seventeenth Americas Conference on Information Systems. 
  90. ^ Brabham, Daren C. (2012). "Motivations for Participation in a Crowdsourcing Application to Improve Public Engagement in Transit Planning". Journal of Applied Communication Research. doi:10.1080/00909882.2012.693940. 
  91. ^ Lietsala, Katri; Joutsen, Atte (2007). "Hang-a-rounds and True Believers: A Case Analysis of the Roles and Motivational Factors of the Star Wreck Fans". MindTrek 2007 Conference Proceedings. 
  92. ^ State of the World’s Volunteerism Report 2011 http://www.unv.org/fileadmin/docdb/pdf/2011/SWVR/English/SWVR2011_full.pdf
  93. ^ Quinn; Bederson (2010). "Human Computation: A Survey and Taxonomy of a Growing Field" (PDF). CHI 2011. 
  94. ^ Aitamurto, Tanja (2011) “The Promise of Idea Crowdsourcing – Benefits, Contexts, Limitations.” Url=http://www.academia.edu/963662/The_Promise_of_Idea_Crowdsourcing_Benefits_Contexts_Limitations).
  95. ^ Aitamurto, Tanja (2011) “The Promise of Idea Crowdsourcing – Benefits, Contexts, Limitations.” url=http://www.academia.edu/963662/The_Promise_of_Idea_Crowdsourcing_Benefits_Contexts_Limitations
  96. ^ Kleeman, Frank (2008) ”Un(der)paid Innovators: The Commercial Utilization of Consumer Work through Crowdsourcing“ url=http://www.sti-studies.de/ojs/index.php/sti/article/view/81/62
  97. ^ Jason (2011) “Crowdsourcing: A Million Heads is Better Than One.” url=http://www.crowdsourcing.org/document/crowdsourcing-a-million-heads-is-better-than-one/8619
  98. ^ Dupree, Steven (2014) “Crowdfunding 101: Pros and Cons” url=http://www.gsb.stanford.edu/ces/crowdfunding-101
  99. ^ Ipeirotis; Provost; Wang (2010). "Quality Management on Amazon Mechanical Turk" (PDF). 
  100. ^ Kurve, Aditya; Miller, David J.; Kesidis, George (30 May 2014). "Multicategory Crowdsourcing Accounting for Variable Task Difficulty, Worker Skill, and Worker Intention". IEEE KDE (99). 
  101. ^ Hirth; Hoßfeld; Tran-Gia (2011), Human Cloud as Emerging Internet Application - Anatomy of the Microworkers Crowdsourcing Platform (PDF) 
  102. ^ Ipeirotis (2010). "Analyzing the Amazon Mechanical Turk Marketplace" (PDF). XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter (ACM) 17 (2). Retrieved February 26, 2012. 
  103. ^ a b Hosaka, Tomoko A. (April 2008). "Facebook asks users to translate for free". MSNBC. 
  104. ^ Britt, Darice. "Crowdsourcing: The Debate Roars On". Retrieved 2012-12-04. 
  105. ^ Woods, Dan (28 September 2009). "The Myth of Crowdsourcing". Forbes. Retrieved 2012-12-04. 
  106. ^ "Fair Labor Standards Act Advisor". Retrieved 28 February 2012. 
  107. ^ Norcie (2011). "Ethical and Practical Considerations for Compensation of Crowdsourced Research Participants". 
  108. ^ Busarovs, Aleksejs (2013). "Ethical Aspects of Crowdsourcing, or is it a Modern Form of Exploitation.". International Journal of Economics & Business Administration 1 (1): 3–14. Retrieved 26 November 2014. 
  109. ^ Paolacci, Chandler (2010). [thttp://journal.sjdm.org/10/10630a/jdm10630a.pdf "Running Experiments on Amazon Mechanical Turk"] (PDF). Judgement and Decision Making 5 (5): 411–419. Retrieved February 26, 2012. 

External links[edit]