Algorithmic bias describes systematic and repeatable errors in a computer system that create unfair outcomes, such as privileging one arbitrary group of users over others. Bias can emerge due to many factors, including but not limited to the design of the algorithm itself, unintended or unanticipated use or decisions relating to the way data is coded, collected, selected or used to train the algorithm. Algorithmic bias is found across platforms, including but not limited to search engine results and social media platforms, and can have impacts ranging from inadvertent privacy violations to reinforcing social biases of race, gender, sexuality, and ethnicity. The study of algorithmic bias is most concerned with algorithms that reflect "systematic and unfair" discrimination. This bias has only recently been addressed in legal frameworks, such as the 2018 European Union's General Data Protection Regulation.
As algorithms expand their ability to organize society, politics, institutions, and behavior, sociologists have become concerned with the ways in which unanticipated output and manipulation of data can impact the physical world. Because algorithms are often considered to be neutral and unbiased, they can inaccurately project greater authority than human expertise, and in some cases, reliance on algorithms can displace human responsibility for their outcomes. Bias can enter into algorithmic systems as a result of pre-existing cultural, social, or institutional expectations; because of technical limitations of their design; or by being used in unanticipated contexts or by audiences who are not considered in the software's initial design.
Algorithmic bias has been cited in cases ranging from election outcomes to the spread of online hate speech. Problems in understanding, researching, and discovering algorithmic bias stem from the proprietary nature of algorithms, which are typically treated as trade secrets. Even when full transparency is provided, the complexity of certain algorithms poses a barrier to understanding their functioning. Furthermore, algorithms may change, or respond to input or output in ways that cannot be anticipated or easily reproduced for analysis. In many cases, even within a single website or application, there is no single "algorithm" to examine, but a network of many interrelated programs and data inputs, even between users of the same service.
- 1 Definitions
- 2 Methods
- 3 History
- 4 Types
- 5 Impact
- 6 Obstacles to research
- 7 Methods and tools
- 8 Regulation
- 9 References
Algorithms are difficult to define, but may be generally understood as sets of instructions that determine how programs read, collect, process, and analyze data to generate output.:13 For a rigorous technical introduction, see Algorithms. Advances in computer hardware have led to an increased ability to process, store and transmit data. This has in turn boosted the design and adoption of technologies such as machine learning and artificial intelligence.:14–15 By analyzing and processing data, algorithms are the backbone of search engines, social media websites, recommendation engines, online retail, online advertising, and more.
Contemporary social scientists are concerned with algorithmic processes embedded into hardware and software applications because of their political and social impact, and question the underlying assumptions of an algorithm's neutrality.:2:563:294 The term algorithmic bias describes systematic and repeatable errors that create unfair outcomes, such as privileging one arbitrary group of users over others. For example, a credit score algorithm may deny a loan without being unfair, if it is consistently weighing relevant financial criteria. If the algorithm recommends loans to one group of users, but denies loans to another set of nearly identical users based on unrelated criteria, and if this behavior can be repeated across multiple occurrences, an algorithm can be described as biased.:332 This bias may be intentional or unintentional.
Bias can be introduced to an algorithm in several ways. During the assemblage of a dataset, data may be collected, digitized, adapted, and entered into a database according to human-designed cataloging criteria.:3 Next, programmers assign priorities, or hierarchies, for how a program assesses and sorts that data. This requires human decisions about how data is categorized, and which data is included or discarded.:4 Some algorithms collect their own data based on human-selected criteria, which can also reflect the bias of human designers.:8 Other algorithms may reinforce stereotypes and preferences as they process and display "relevant" data for human users, for example, by selecting information based on previous choices of a similar user or group of users.:6
Beyond assembling and processing data, bias can emerge as a result of design. For example, algorithms that determine the allocation of resources or scrutiny (such as determining school placements) may inadvertently discriminate against a category when determining risk based on similar users (as in credit scores).:36 Meanwhile, recommendation engines that work by associating users with similar users, or that make use of inferred marketing traits, might rely on inaccurate associations that reflect broad ethnic, gender, socio-economic, or racial stereotypes. Another example comes from determining criteria for what is included and excluded from results. This criteria could present unanticipated outcomes for search results, such as in flight-recommendation software that omits flights that do not follow the sponsoring airline's flight paths. Algorithms may also display an uncertainty bias, offering more confident assessments when larger data sets are available. This can skew algorithmic processes toward results that more closely correspond with larger samples, which may disregard data from underrepresented populations.:4
The earliest computer programs were designed to mimic human reasoning and deductions, and were deemed to be functioning when they successfully and consistently reproduced that human logic. In his 1976 book Computer Power and Human Reason, Artificial Intelligence pioneer Joseph Weizenbaum suggested that bias could arise both from the data used in a program, but also from the way a program is coded.:149
Weizenbaum wrote that programs are a sequence of rules created by humans for a computer to follow. By following those rules consistently, such programs "embody law,":40 that is, enforce a specific way to solve problems. The rules a computer follows are based on the assumptions of a computer programmer for how these problems might be solved. That means the code could incorporate the programmer's imagination of how the world works, including his or her biases and expectations.:109 While a computer program can incorporate bias in this way, Weizenbaum also noted that any data fed to a machine additionally reflects "human decisionmaking processes" as data is being selected.:70, 105
Finally, he noted that machines might also transfer good information with unintended consequences if users are unclear about how to interpret the results.:65 Weizenbaum warned against trusting decisions made by computer programs that a user doesn't understand, comparing such faith to a tourist who can find his way to a hotel room exclusively by turning left or right on a coin toss. Crucially, the tourist has no basis of understanding how or why he arrived at his destination, and a successful arrival does not mean the process is accurate or reliable.:226
An early example of algorithmic bias resulted in as many as 60 women and ethnic minorities denied entry to St. George's Hospital Medical School per year from 1982 to 1986, based on implementation of a new computer-guidance assessment system that denied entry to women and men with "foreign-sounding names" based on historical trends in admissions.
Contemporary critiques and responses
Though well-designed algorithms frequently determine outcomes that are equally (or more) equitable than the decisions of human beings, cases of bias still occur, and are difficult to predict and analyze. The complexity of analyzing algorithmic bias has grown alongside the complexity of programs and their design. Decisions made by one designer, or team of designers, may be obscured among the many pieces of code created for a single program; over time these decisions and their collective impact on the program's output may be forgotten.:115 In theory, these biases may create new patterns of behavior, or "scripts," in relationship to specific technologies as the code interacts with other elements of society. Biases may also impact how society shapes itself around the data points that algorithms require. For example, if data shows a high number of arrests in a particular area, an algorithm may assign more police patrols to that area, which could lead to more arrests.:180
The decisions of algorithmic programs can be seen as more authoritative than the decisions of the human beings they are meant to assist,:15 a process described by author Clay Shirky as "algorithmic authority". Shirky uses the term to describe "the decision to regard as authoritative an unmanaged process of extracting value from diverse, untrustworthy sources," such as search results. This neutrality can also be misrepresented by the language used by experts and the media when results are presented to the public. For example, a list of news items selected and presented as "trending" or "popular" may be created based on significantly wider criteria than just their popularity.:14
Because of their convenience and authority, algorithms are theorized as a means of delegating responsibility away from humans.:16:6 This can have the effect of reducing alternative options, compromises, or flexibility.:16 Sociologist Scott Lash has critiqued algorithms as a new form of "generative power", in that they are a virtual means of generating actual ends. Where previously human behavior generated data to be collected and studied, powerful algorithms increasingly could shape and define human behaviors.:71
Concerns over the impact of algorithms on society have led to the creation of working groups in organizations such as Google and Microsoft, which have co-created a working group named Fairness, Accountability, and Transparency in Machine Learning.:115 Ideas from Google have included community groups that patrol the outcomes of algorithms and vote to control or restrict outputs they deem to have negative consequences.:117 In recent years, the study of the Fairness, Accountability, and Transparency (FAT) of algorithms has emerged as its own interdisciplinary research area with an annual conference called FAT*.
Pre-existing bias in an algorithm is a consequence of underlying social and institutional ideologies. Such ideas may influence or create personal biases within individual designers or programmers. Such prejudices can be explicit and conscious, or implicit and unconscious.:334:294 Poorly selected input data will influence the outcomes created by machines.:17 Encoding pre-existing bias into software can preserve social and institutional bias, and without correction, could be replicated in all future uses of that algorithm.:116:8
An example of this form of bias is the British Nationality Act Program, designed to automate the evaluation of new UK citizens after the 1981 British Nationality Act.:341 The program accurately reflected the tenets of the law, which stated that "a man is the father of only his legitimate children, whereas a woman is the mother of all her children, legitimate or not.":341:375 In its attempt to transfer a particular logic into an algorithmic process, the BNAP inscribed the logic of the British Nationality Act into its algorithm, which would perpetuate it even if the act was eventually repealed.:342
Technical bias emerges through limitations of a program, computational power, its design, or other constraint on the system.:332 Such bias can also be a restraint of design, for example, a search engine that shows three results per screen can be understood to privilege the top three results slightly more than the next three, as in an airline price display.:336 Another case is software that relies on randomness for fair distributions of results. If the random number generation mechanism is not truly random, it can introduce bias, for example, by skewing selections toward items at the end or beginning of a list.:332
A decontextualized algorithm uses unrelated information to sort results, for example, a flight-pricing algorithm that sorts results by alphabetical order would be biased in favor of American Airlines over United Airlines.:332 The opposite may also apply, in which results are evaluated in contexts different from which they are collected. Data may be collected without crucial external context: for example, when facial recognition software is used by surveillance cameras, but evaluated by remote staff in another country or region, or evaluated by non-human algorithms with no awareness of what takes place beyond the camera's field of vision. This could create an incomplete understanding of a crime scene, for example, potentially mistaking bystanders for those who commit the crime.:574
Lastly, technical bias can be created by attempting to formalize decisions into concrete steps on the assumption that human behavior works in the same way. For example, software weighs data points to determine whether a defendant should accept a plea bargain, while ignoring the impact of emotion on a jury.:332 Another unintended result of this form of bias was found in the plagiarism-detection software Turnitin, which compares student-written texts to information found online and returns a probability score that the student's work is copied. Because the software compares long strings of text, it is more likely to identify non-native speakers of English than native speakers, as the latter group might be better able to change individual words, break up strings of plagiarized text, or obscure copied passages through synonyms. Because it is easier for native speakers to evade detection as a result of the technical constraints of the software, this creates a scenario where Turnitin identifies foreign-speakers of English for plagiarism while allowing more native-speakers to evade detection.:21–22
Emergent bias is the result of the use and reliance on algorithms across new or unanticipated contexts.:334 Algorithms may not have been adjusted to consider new forms of knowledge, such as new drugs or medical breakthroughs, new laws, business models, or shifting cultural norms.:334,336 This may exclude groups through technology, without providing clear outlines to understand who is responsible for their exclusion.:179:294 Similarly, problems may emerge when training data (the samples "fed" to a machine, by which it models certain conclusions) do not align with contexts that an algorithm encounters in the real world.
In 1990, an example of emergent bias was identified in the software used to place US medical students into residencies, the National Residency Match Program (NRMP).:338 The algorithm was designed at a time when few married couples would seek residencies together. As more women entered medical schools, more students were likely to request a residency alongside their partners. The process called for each applicant to provide a list of preferences for placement across the US, which was then sorted and assigned when a hospital and an applicant both agreed to a match. In the case of married couples where both sought residencies, the algorithm weighed the location choices of the higher-rated partner first. The result was a frequent assignment of highly preferred schools to the first partner and lower-preferred schools to the second partner, rather than sorting for compromises in placement preference.:338
Additional emergent biases include:
Unpredictable correlations can emerge when large data sets are compared to each other. For example, data collected about web-browsing patterns may align with signals marking sensitive data (such as race or sexual orientation). By selecting according to certain behavior or browsing patterns, the end effect would be almost identical to discrimination through the use of direct race or sexual orientation data.:6 In other cases, the algorithm draws conclusions from correlations, without being able to understand those correlations. For example, one triage program gave lower priority to asthmatics who had pneumonia than asthmatics who did not have pneumonia. The program algorithm did this because it simply compared survival rates: asthmatics with pneumonia are at the highest risk. Historically, for this same reason, hospitals typically give such asthmatics the best and most immediate care.
Emergent bias can occur when an algorithm is used by unanticipated audiences. For example, machines may require that users can read, write, or understand numbers, or relate to an interface using metaphors that they do not understand.:334 These exclusions can become compounded, as biased or exclusionary technology is more deeply integrated into society.:179
Apart from exclusion, unanticipated uses may emerge from the end user relying on the software rather than their own knowledge. In one example, an unanticipated user group led to algorithmic bias in the UK, when the British National Act Program was created as a proof-of-concept by computer scientists and immigration lawyers to evaluate suitability for British citizenship. The designers had access to legal expertise beyond the end users in immigration offices, whose understanding of both software and immigration law would likely have been unsophisticated. The agents administering the questions relied entirely on the software, which excluded alternative pathways to citizenship, and used the software even after new case laws and legal interpretations led the algorithm to become outdated. As a result of designing an algorithm for users assumed to be legally savvy on immigration law, the software's algorithm indirectly led to bias in favor of applicants who fit a very narrow set of legal criteria set by the algorithm, rather than by the more broader criteria of UK immigration law.:342
Emergent bias may also create a feedback loop, or recursion, if data collected for an algorithm results in real-world responses which are fed back into the algorithm. For example, simulations of the predictive policing software (PredPol), deployed in Oakland, California, suggested an increased police presence in black neighborhoods based on crime data reported by the public. The simulation showed that the public reported crime based on the sight of police cars, regardless of what police were doing. The simulation interpreted police car sightings in modeling its predictions of crime, and would in turn assign an even larger increase of police presence within those neighborhoods. The Human Rights Data Analysis Group, which conducted the simulation, warned that in places where racial discrimination is a factor in arrests, such feedback loops could reinforce and perpetuate racial discrimination in policing.
Recommender systems such as those used to recommend online videos or news articles can create feedback loops. When users click on content that is suggested by algorithms, it influences the next set of suggestions. Over time this may lead to users entering a Filter Bubble and being unaware of important or useful content.
Corporate algorithms could be skewed to invisibly favor financial arrangements or agreements between companies, without the knowledge of a user who may mistake the algorithm as being impartial. For example, American Airlines created a flight-finding algorithm in the 1980s. The software presented a range of flights from various airlines to customers, but weighed factors that boosted its own flights, regardless of price or convenience. In testimony to the United States Congress, the president of the airline stated outright that the system was created with the intention of gaining competitive advantage through preferential treatment.:2:331
In a 1998 paper describing Google, it was shown that the founders of the company adopted a policy of transparency in search results regarding paid placement, arguing that "advertising-funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers." This bias would be an "invisible" manipulation of the user.:3
A series of studies about undecided voters in the US and in India found that search engine results were able to shift voting outcomes by about 20%. The researchers concluded that candidates have "no means of competing" if an algorithm, with or without intent, boosted page listings for a rival candidate. Facebook users who saw messages related to voting were more likely to vote. A 2010 randomized trial of Facebook users showed a 20% increase (340,000 votes) among users who saw messages encouraging voting, as well as images of their friends who had voted. Legal scholar Jonathan Zittrain has warned that this could create a "digital gerrymandering" effect in elections, "the selective presentation of information by an intermediary to meet its agenda, rather than to serve its users", if intentionally manipulated.:335
In 2016, the professional networking site LinkedIn was discovered to recommend male variations of women's names in response to search queries. The site did not make similar recommendations in searches for male names. For example, "Andrea" would bring up a prompt asking if users meant "Andrew," but queries for "Andrew" did not ask if users meant to find "Andrea". The company said this was the result of an analysis of users' interactions with the site.
In 2012, the department store franchise Target was cited for gathering data points to infer when women customers were pregnant, even if they had not announced it, and then sharing that information with marketing partners.:94 Because the data had been predicted, rather than directly observed or reported, the company had no legal obligation to protect the privacy of those customers.:98
Web search algorithms have also been accused of bias. Google's results may prioritize pornographic content in search terms related to sexuality, for example, "lesbian". This bias extends to the search engine showing popular but sexualized content in neutral searches. For example, "Top 25 Sexiest Women Athletes" articles displayed as first-page results in searches for "women athletes".:31 In 2017, Google adjusted these results along with others that surfaced hate groups, racist views, child abuse and pornography, and other upsetting and offensive content. Other examples include the display of higher-paying jobs to male applicants on job search websites.
Racial and ethnic discrimination
Algorithms have been criticized as a method for obscuring racial prejudices in decision-making.:158 Because of how certain races and ethnic groups were treated in the past, data can often contain hidden biases. For example, black people are likely to receive longer sentences than white people who committed the same crime. This could potentially mean that a system amplifies the original biases in the data.
One example is the use of risk assessments in criminal sentencing in the United States and parole hearings, judges were presented with an algorithmically generated score intended to reflect the risk that a prisoner will repeat a crime. For the time period starting in 1920 and ending in 1970, the nationality of a criminals's father was a consideration in those risk assessment scores.:4 Today, these scores are shared with judges in Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington, and Wisconsin. An independent investigation by ProPublica found that the scores were inaccurate 80% of the time, and disproportionately skewed to suggest blacks to be at risk of relapse, 77% more often than whites.
In 2015, Google apologized when black users complained that an image-identification algorithm in its Photos application identified them as gorillas. In 2010, Nikon cameras were criticized when image-recognition algorithms consistently asked Asian users if they were blinking. Such examples are the product of bias in biometric data sets. Biometric data is drawn from aspects of the body, including racial features either observed or inferred, which can then be transferred into data points.:154 Speech recognition technology can have different accuracies depending on the user's accent. This may be caused by the a lack of training data for speakers of that accent.
Biometric data about race may also be inferred, rather than observed. For example, a 2012 study showed that names commonly associated with blacks were more likely to yield search results implying arrest records, regardless of whether there is any police record of that individual's name.
One study that set out to examine “Risk, Race, & Recidivism: Predictive Bias and Disparate Impact” alleges a two-fold (45 percent vs. 23 percent) adverse likelihood for black vs. Caucasian defendants to be misclassified as imposing a higher risk despite having objectively remained without any documented recidivism over a two-year period of observation. .
Online hate speech
In 2017 a Facebook algorithm designed to remove online hate speech was found to advantage white men over black children when assessing objectionable content, according to internal Facebook documents. The algorithm, which is a combination of computer programs and human content reviewers, was created to protect broad categories rather than specific subsets of categories. For example, posts denouncing "Muslims" would be blocked, while posts denouncing "Radical Muslims" would be allowed. An unanticipated outcome of the algorithm is to allow hate speech against black children, because they denounce the "children" subset of blacks, rather than "all blacks," whereas "all white men" would trigger a block, because whites and males are not considered subsets. Facebook was also found to allow ad purchasers to target "Jew haters" as a category of users, which the company said was an inadvertent outcome of algorithms used in assessing and categorizing data. The company's design also allowed ad buyers to block African-Americans from seeing housing ads.
Surveillance camera software may be considered inherently political because it requires algorithms to distinguish normal from abnormal behaviors, and to determine who belongs in certain locations at certain times.:572 The ability of such algorithms to recognize faces across a racial spectrum has been shown to be limited by the racial diversity of images in its training database; if the majority of photos belong to one race or gender, the software is better at recognizing other members of that race or gender. A 2002 analysis of software used to identify individuals in CCTV images found several examples of bias when run against criminal databases. The software was assessed as identifying men more frequently than women, older people more frequently than the young, and identified Asians, African-Americans and other races more often than whites.:190 Additional studies of facial recognition software have found the opposite to be true when trained on non-criminal databases, with the software being the least accurate in identifying darker-skinned females.
In 2011, users of the gay hookup application Grindr reported that the Android store's recommendation algorithm was linking Grindr to applications designed to find sex offenders, which critics said inaccurately related homosexuality with pedophilia. Writer Mike Ananny criticized this association in The Atlantic, arguing that such associations further stigmatized gay men. In 2009, online retailer Amazon de-listed 57,000 books after an algorithmic change expanded its "adult content" blacklist to include any book addressing sexuality or gay themes, such as the critically acclaimed novel Brokeback Mountain.:5
In 2019, it was found that on Facebook, searches for "photos of my female friends" yielded suggestions such as "in bikinis" or "at the beach". In contrast, searches for "photos of my male friends" yielded no results.
Obstacles to research
Several problems impede the study of large-scale algorithmic bias, hindering the application of academically rigorous studies and public understanding.:5
Lack of transparency
Commercial algorithms are proprietary, and may be treated as trade secrets.:2:7:183 Treating algorithms as trade secrets protects companies, such as search engines, where a transparent algorithm might reveal tactics to manipulate search rankings.:366 This makes it difficult for researchers to conduct interviews or analysis to discover how algorithms function.:20 Critics suggest that such secrecy can also obscure possible unethical methods used in producing or processing algorithmic output.:369
Algorithmic processes are complex, often exceeding the understanding of the people who use them.:2:7 Large-scale operations may not be understood even by those involved in creating them. The methods and processes of contemporary programs are often obscured by the inability to know every permutation of a code's input or output.:183
Social scientist Bruno Latour has identified this process as blackboxing, a process in which "scientific and technical work is made invisible by its own success. When a machine runs efficiently, when a matter of fact is settled, one need focus only on its inputs and outputs and not on its internal complexity. Thus, paradoxically, the more science and technology succeed, the more opaque and obscure they become." Others have critiqued the black box metaphor, suggesting that current algorithms are not one black box, but a network of interconnected ones.:92
An example of this complexity can be found in the range of inputs into customizing feedback. The social media site Facebook factored in at least 100,000 data points to determine the layout of a user's social media feed in 2013. Furthermore, large teams of programmers may operate in relative isolation from one another, and be unaware of the cumulative effects of small decisions within connected, elaborate algorithms.:118 Not all code is original, and may be borrowed from other libraries, creating a complicated set of relationships between data processing and data input systems.:22
Additional complexity occurs through machine learning and the personalization of algorithms based on user interactions such as clicks, time spent on site, and other metrics. These personal adjustments can confuse general attempts to understand algorithms.:367:7 One unidentified streaming radio service reported that it used five unique music-selection algorithms it selected for its users, based on their behavior. This creates different experiences of the same streaming services between different users, making it harder to understand what these algorithms do.:5 Companies also run frequent A/B tests to fine-tune algorithms based on user response. For example, the search engine Bing can run up to ten million subtle variations of its service per day, creating different experiences of the service between each use and/or user.:5
Lack of data about sensitive categories
A significant barrier to understanding the tackling of bias in practice is that categories, such as demographics of individuals protected by anti-discrimination law, are often not explicitly considered when collecting and processing data. In some cases, there is little opportunity to collect this data explicitly, such as in device fingerprinting, ubiquitous computing and the Internet of Things. In other cases, the data controller may not wish to collect such data for reputational reasons, or because it represents a heightened liability and security risk. It may also be the case that, at least in relation to the European Union's General Data Protection Regulation, such data falls under the 'special category' provisions (Article 9), and therefore comes with more restrictions on potential collection and processing.
Some practitioners have tried to estimate and impute these missing sensitive categorisations in order to allow bias mitigation, for example building systems to infer ethnicity from names, however this can introduce other forms of bias if not undertaken with care. Machine learning researchers have drawn upon cryptographic privacy-enhancing technologies such as secure multi-party computation to propose methods whereby algorithmic bias can be assessed or mitigated without these data ever being available to modellers in cleartext.
Algorithmic bias does not only include protected categories, but can also concerns characteristics less easily observable or codifiable, such as political viewpoints. In these cases, there is rarely an easily accessible or non-controversial ground truth, and removing the bias from such a system is more difficult.
Furthermore, false and accidental correlations can emerge from a lack of understanding of protected categories, for example, insurance rates based on historical data of car accidents which may overlap, strictly by coincidence, with residential clusters of ethnic minorities.
Methods and tools
There have been several attempts to create methods and tools that can detect and observe biases within an algorithm. These emergent field focuses on tools which are typically applied to the (training) data used by the program rather than the algorithm's internal processes. These methods may also analyze a program's output and its usefulness and therefore may involve the analysis of its confusion matrix (or table of confusion).
Currently, a new IEEE standard is being drafted that aims to specify methodologies which help creators of algorithms eliminate issues of bias and articulate transparency (i.e. to authorities or end users) about the function and possible effects of their algorithms. The project was approved February 2017 and is sponsored by the Software & Systems Engineering Standards Committee, a committee chartered by the IEEE Computer Society. A draft of the standard is expected to be submitted for balloting in June 2019.
The General Data Protection Regulation (GDPR), the European Union's revised data protection regime that was implemented in 2018, addresses "Automated individual decision-making, including profiling" in Article 22. These rules prohibit "solely" automated decisions which have a "significant" or "legal" effect on an individual, unless they are explicitly authorised by consent, contract, or member state law. Where they are permitted, there must be safeguards in place, such as a right to a human-in-the-loop, and a non-binding right to an explanation of decisions reached. While these regulations are commonly considered to be new, nearly identical provisions have existed across Europe since 1995, in Article 15 of the Data Protection Directive. The original automated decision rules and safeguards found in French law since the late 1970s.
... the controller should use appropriate mathematical or statistical procedures for the profiling, implement technical and organisational measures appropriate ... that prevents, inter alia, discriminatory effects on natural persons on the basis of racial or ethnic origin, political opinion, religion or beliefs, trade union membership, genetic or health status or sexual orientation, or that result in measures having such an effect.
Like the non-binding right to an explanation in recital 71, the problem is the non-binding nature of recitals. While it has been treated as a requirement by the Article 29 Working Party that advised on the implementation of data protection law, its practical dimensions are unclear. It has been argued that the Data Protection Impact Assessments for high risk data profiling (alongside other pre-emptive measures within data protection) may be a better way to tackle issues of algorithmic discrimination, as it restricts the actions of those deploying algorithms, rather than requiring consumers to file complaints or request changes.
The United States has no general legislation controlling algorithmic bias, approaching the problem through various state and federal laws that might vary by industry, sector, and by how an algorithm is used. Many policies are self-enforced or controlled by the Federal Trade Commission. In 2016, the Obama administration released the National Artificial Intelligence Research and Development Strategic Plan, which was intended to guide policymakers toward a critical assessment of algorithms. It recommended researchers to "design these systems so that their actions and decision-making are transparent and easily interpretable by humans, and thus can be examined for any bias they may contain, rather than just learning and repeating these biases". Intended only as guidance, the report did not create any legal precedent.:26
In 2017, New York City passed the first algorithmic accountability bill in the United States. The bill, which went into effect on January 1, 2018, required "the creation of a task force that provides recommendations on how information on agency automated decision systems may be shared with the public, and how agencies may address instances where people are harmed by agency automated decision systems." The task force is required to present findings and recommendations for further regulatory action in 2019.
On July 31, 2018, a draft of the Personal Data Bill was presented. The draft proposes standards for the storage, processing and transmission of data. While it does not use the term algorithm, it makes for provisions for "...harm resulting from any processing or any kind of processing undertaken by the fiduciary". It defines "any denial or withdrawal of a service, benefit or good resulting from an evaluative decision about the data principal" or "any discriminatory treatment" as a source of harm that could arise from improper use of data. It also makes special provisions for people of "Intersex status”.
- Jacobi, Jennifer (13 September 2001). "Patent #US2001021914". Espacenet. Retrieved 4 July 2018.
- Striphas, Ted. "What is an Algorithm? – Culture Digitally". culturedigitally.org. Retrieved 20 November 2017.
- Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009). Introduction to algorithms (3rd ed.). Cambridge, Mass.: MIT Press. p. 5. ISBN 978-0-262-03384-8.
- Kitchin, Rob (25 February 2016). "Thinking critically about and researching algorithms" (PDF). Information, Communication & Society. 20 (1): 14–29. doi:10.1080/1369118X.2016.1154087. Retrieved 19 November 2017.
- Google. "How Google Search Works". Retrieved 19 November 2017.
- Luckerson, Victor. "Here's How Your Facebook News Feed Actually Works". TIME.com. Retrieved 19 November 2017.
- Vanderbilt, Tom (2013-08-07). "The Science Behind the Netflix Algorithms That Decide What You'll Watch Next". Wired. Retrieved 19 November 2017.
- Angwin, Julia; Mattu, Surya (20 September 2016). "Amazon Says It Puts Customers First. But Its Pricing Algorithm Doesn't — ProPublica". ProPublica. Retrieved 19 November 2017.
- Livingstone, Rob. "The future of online advertising is big data and algorithms". The Conversation. Retrieved 19 November 2017.
- Hickman, Leo (1 July 2013). "How algorithms rule the world". The Guardian. Retrieved 19 November 2017.
- Seaver, Nick. "Knowing Algorithms" (PDF). Media in Transition 8, Cambridge, MA, April 2013. Retrieved 18 November 2017.
- Graham, Stephen D.N. (July 2016). "Software-sorted geographies" (PDF). Progress in Human Geography (Submitted manuscript). 29 (5): 562–580. doi:10.1191/0309132505ph568oa.
- Tewell, Eamon (4 April 2016). "Toward the Resistant Reading of Information: Google, Resistant Spectatorship, and Critical Information Literacy". Portal: Libraries and the Academy. 16 (2): 289–310. ISSN 1530-7131. Retrieved 19 November 2017.
- Crawford, Kate (1 April 2013). "The Hidden Biases in Big Data". Harvard Business Review.
- Friedman, Batya; Nissenbaum, Helen (July 1996). "Bias in Computer Systems" (PDF). ACM Transactions on Information Systems. 14 (3): 330–347. Retrieved 10 March 2019.
- Gillespie, Tarleton; Boczkowski, Pablo; Foot, Kristin (2014). Media Technologies. Cambridge: MIT Press. pp. 1–30. ISBN 9780262525374.
- Diakopoulas, Nicholas. "Algorithmic Accountability: On the Investigation of Black Boxes". towcenter.org. Retrieved 19 November 2017.
- Lipartito, Kenneth (6 January 2011). "The Narrative and the Algorithm: Genres of Credit Reporting from the Nineteenth Century to Today" (PDF). SSRN Electronic Journal (Submitted manuscript). doi:10.2139/ssrn.1736283.
- Goodman, Bryce; Flaxman, Seth (2017). "EU regulations on algorithmic decision-making and a "right to explanation"". AI Magazine. 38 (3): 50. arXiv:1606.08813. doi:10.1609/aimag.v38i3.2741.
- Weizenbaum, Joseph (1976). Computer power and human reason : from judgment to calculation. San Francisco: W.H. Freeman. ISBN 978-0-7167-0464-5.
- Goffrey, Andrew (2008). "Algorithm". In Fuller, Matthew (ed.). Software studies: a lexicon. Cambridge, Mass.: MIT Press. pp. 15–20. ISBN 978-1-4356-4787-9.
- Lowry, Stella; Macpherson, Gordon (5 March 1988). "A Blot on the Profession". British Medical Journal. 296 (6623): 657. Retrieved 17 November 2017.
- Miller, Alex P. (26 July 2018). "Want Less-Biased Decisions? Use Algorithms". Harvard Business Review. Retrieved 31 July 2018.
- Introna, Lucas D. (2 December 2011). "The Enframing of Code". Theory, Culture & Society. 28 (6): 113–141. doi:10.1177/0263276411418131.
- Bogost, Ian (2015-01-15). "The Cathedral of Computation". The Atlantic. Retrieved 19 November 2017.
- Introna, Lucas; Wood, David (2004). "Picturing algorithmic surveillance: the politics of facial recognition systems". Surveillance & Society. 2: 177–198. Retrieved 19 November 2017.
- Introna, Lucas D. (21 December 2006). "Maintaining the reversibility of foldings: Making the ethics (politics) of information technology visible". Ethics and Information Technology. 9 (1): 11–25. CiteSeerX 10.1.1.154.1313. doi:10.1007/s10676-006-9133-z.
- Shirky, Clay. "A Speculative Post on the Idea of Algorithmic Authority Clay Shirky". www.shirky.com. Retrieved 20 November 2017.
- Ziewitz, Malte (1 January 2016). "Governing Algorithms: Myth, Mess, and Methods". Science, Technology, & Human Values. 41 (1): 3–16. doi:10.1177/0162243915608948. ISSN 0162-2439. Retrieved 22 November 2017.
- Lash, Scott (30 June 2016). "Power after Hegemony". Theory, Culture & Society. 24 (3): 55–78. doi:10.1177/0263276407075956.
- Garcia, Megan (1 January 2016). "Racist in the Machine". World Policy Journal. 33 (4): 111–117. doi:10.1215/07402775-3813015.
- "ACM FAT* - 2018 Information for Press". fatconference.org. Retrieved 2019-02-26.
- Sergot, MJ; Sadri, F; Kowalski, RA; Kriwaczek, F; Hammond, P; Cory, HT (May 1986). "The British Nationality Act as a Logic Program" (PDF). Communications of the ACM. 29 (5): 370–386. Retrieved 18 November 2017.
- Gillespie, Tarleton. "Algorithm [draft] [#digitalkeywords] – Culture Digitally". culturedigitally.org. Retrieved 20 November 2017.
- Roth, A. E. 1524–1528. (14 December 1990). "New physicians: A natural experiment in market organization". Science. 250 (4987): 1524–1528. Bibcode:1990Sci...250.1524R. doi:10.1126/science.2274783. Retrieved 18 November 2017.
- Kuang, Cliff (21 November 2017). "Can A.I. Be Taught to Explain Itself?". The New York Times. Retrieved 26 November 2017.
- Jouvenal, Justin (17 November 2016). "Police are using software to predict crime. Is it a 'holy grail' or biased against minorities?". Washington Post. Retrieved 25 November 2017.
- Chamma, Maurice (2016-02-03). "Policing the Future". The Marshall Project. Retrieved 25 November 2017.
- Lum, Kristian; Isaac, William (October 2016). "To predict and serve?". Significance. 13 (5): 14–19. doi:10.1111/j.1740-9713.2016.00960.x.
- Smith, Jack. "Predictive policing only amplifies racial bias, study shows". Mic. Retrieved 25 November 2017.
- Lum, Kristian; Isaac, William (1 October 2016). "FAQs on Predictive Policing and Bias". HRDAG. Retrieved 25 November 2017.
- Sinha, Ayan; Gleich, David F.; Ramani, Karthik (2018-08-09). "Gauss's law for networks directly reveals community boundaries". Scientific Reports. 8 (1). doi:10.1038/s41598-018-30401-0. ISSN 2045-2322.
- Hao, Karen; Hao, Karen. "Google is finally admitting it has a filter-bubble problem". Quartz. Retrieved 2019-02-26.
- "Facebook Is Testing This New Feature to Fight 'Filter Bubbles'". Fortune. Retrieved 2019-02-26.
- Sandvig, Christian; Hamilton, Kevin; Karahalios, Karrie; Langbort, Cedric (22 May 2014). "Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms" (PDF). 64th Annual Meeting of the International Communication Association. Retrieved 18 November 2017.
- Brin, Sergey; Page, Lawrence. "The Anatomy of a Search Engine". www7.scu.edu.au. Retrieved 18 November 2017.
- Epstein, Robert; Robertson, Ronald E. (18 August 2015). "The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections". Proceedings of the National Academy of Sciences. 112 (33): E4512–E4521. Bibcode:2015PNAS..112E4512E. doi:10.1073/pnas.1419828112. PMC 4547273. PMID 26243876. Retrieved 19 November 2017.
- Bond, Robert M.; Fariss, Christopher J.; Jones, Jason J.; Kramer, Adam D. I.; Marlow, Cameron; Settle, Jaime E.; Fowler, James H. (13 September 2012). "A 61-million-person experiment in social influence and political mobilization". Nature. 489 (7415): 295–8. Bibcode:2012Natur.489..295B. doi:10.1038/nature11421. ISSN 0028-0836. PMC 3834737. PMID 22972300.
- Zittrain, Jonathan (2014). "Engineering an Election" (PDF). Harvard Law Review Forum. 127: 335–341. Retrieved 19 November 2017.
- Day, Matt (31 August 2016). "How LinkedIn's search engine may reflect a gender bias". The Seattle Times. Retrieved 25 November 2017.
- Crawford, Kate; Schultz, Jason (2014). "Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms". Boston College Law Review. 55 (1): 93–128. Retrieved 18 November 2017.
- Duhigg, Charles (16 February 2012). "How Companies Learn Your Secrets". The New York Times. Retrieved 18 November 2017.
- Noble, Safiya (2012). "Missed Connections: What Search Engines Say about Women" (PDF). Bitch Magazine. 12 (4): 37–41.
- Guynn, Jessica (16 March 2017). "Google starts flagging offensive content in search results". USA TODAY. USA Today. Retrieved 19 November 2017.
- Simonite, Tom. "Study Suggests Google's Ad-Targeting System May Discriminate". MIT Technology Review. Massachusetts Institute of Technology. Retrieved 17 November 2017.
- Dastin, Jeffrey (October 9, 2018). "Amazon scraps secret AI recruiting tool that showed bias against women". Reuters.
- Nakamura, Lisa (2009). Magnet, Shoshana; Gates, Kelly (eds.). The new media of surveillance. London: Routledge. pp. 149–162. ISBN 978-0-415-56812-8.
- Alexander, Rudolph; Gyamerah, Jacquelyn (September 1997). "Differential Punishing of African Americans and Whites Who Possess Drugs: A Just Policy or a Continuation of the Past?". Journal of Black Studies. 28 (1): 97–111. doi:10.1177/002193479702800106. ISSN 0021-9347.
- Petersilia, Joan (January 1985). "Racial Disparities in the Criminal Justice System: A Summary". Crime & Delinquency. 31 (1): 15–34. doi:10.1177/0011128785031001002. ISSN 0011-1287.
- Angwin, Julia; Larson, Jeff; Mattu, Surya; Kirchner, Lauren (23 May 2016). "Machine Bias — ProPublica". ProPublica. Retrieved 18 November 2017.
- Harcourt, Bernard (16 September 2010). "Risk as a Proxy for Race". Criminology and Public Policy, Forthcoming. SSRN 1677654.
- Guynn, Jessica (1 July 2015). "Google Photos labeled black people 'gorillas'". USA TODAY. USA Today. USA Today. Retrieved 18 November 2017.
- Rose, Adam (22 January 2010). "Are Face-Detection Cameras Racist?". Time. Retrieved 18 November 2017.
- "Alexa does not understand your accent". Washington Post.
- Sweeney, Latanya (28 January 2013). "Discrimination in Online Ad Delivery". SSRI. doi:10.2139/ssrn.2208240.
- Skeem J, Lowenkamp C, Risk, Race, & Recidivism: Predictive Bias and Disparate Impact, (June 14, 2016). Available at SSRN: https://ssrn.com/abstract=2687339 or http://dx.doi.org/10.2139/ssrn.2687339
- Angwin, Julia; Grassegger, Hannes (28 June 2017). "Facebook's Secret Censorship Rules Protect White Men From Hate Speech But Not Black Children — ProPublica". ProPublica. Retrieved 20 November 2017.
- Angwin, Julia; Varner, Madeleine; Tobin, Ariana (14 September 2017). "Facebook Enabled Advertisers to Reach 'Jew Haters' — ProPublica". ProPublica. Retrieved 20 November 2017.
- Furl, N (December 2002). "Face recognition algorithms and the other-race effect: computational mechanisms for a developmental contact hypothesis". Cognitive Science. 26 (6): 797–815. doi:10.1016/S0364-0213(02)00084-8.
- Buolamwini, Joy; Gebru, Timnit (2018). "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification" (PDF). Proceedings of Machine Learning Research. 81: 1 – via MLR Press.
- Ananny, Mike (2011-04-14). "The Curious Connection Between Apps for Gay Men and Sex Offenders". The Atlantic. Retrieved 18 November 2017.
- Kafka, Peter. "Did Amazon Really Fail This Weekend? The Twittersphere Says 'Yes,' Online Retailer Says 'Glitch.'". AllThingsD. Retrieved 22 November 2017.
- Kafka, Peter. "Amazon Apologizes for 'Ham-fisted Cataloging Error'". AllThingsD. AllThingsD. Retrieved 22 November 2017.
- Matsakis, Louise (2019-02-22). "A 'Sexist' Search Bug Says More About Us Than Facebook". Wired. ISSN 1059-1028. Retrieved 2019-02-26.
- Sandvig, Christian; Hamilton, Kevin; Karahalios, Karrie; Langbort, Cedric (2014). Gangadharan, Seeta Pena; Eubanks, Virginia; Barocas, Solon (eds.). "An Algorithm Audit" (PDF). Data and Discrimination: Collected Essays.
- Granka, Laura A. (27 September 2010). "The Politics of Search: A Decade Retrospective" (PDF). The Information Society. 26 (5): 364–374. doi:10.1080/01972243.2010.511560. Retrieved 18 November 2017.
- Kitchin, Rob (25 February 2016). "Thinking critically about and researching algorithms". Information, Communication & Society. 20 (1): 14–29. doi:10.1080/1369118X.2016.1154087.
- LaFrance, Adrienne (2015-09-18). "The Algorithms That Power the Web Are Only Getting More Mysterious". The Atlantic. Retrieved 19 November 2017.
- Bruno Latour (1999). Pandora's hope: essays on the reality of science studies. Cambridge, Massachusetts: Harvard University Press.
- Kubitschko, Sebastian; Kaun, Anne (2016). Innovative Methods in Media and Communication Research. Springer. ISBN 978-3-319-40700-5. Retrieved 19 November 2017.
- McGee, Matt (16 August 2013). "EdgeRank Is Dead: Facebook's News Feed Algorithm Now Has Close To 100K Weight Factors". Marketing Land. Retrieved 18 November 2017.
- Veale, Michael; Binns, Reuben (2017). "Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data". Big Data & Society. 4 (2): 205395171774353. doi:10.1177/2053951717743530. SSRN 3060763.
- Elliott, Marc N.; Morrison, Peter A.; Fremont, Allen; McCaffrey, Daniel F.; Pantoja, Philip; Lurie, Nicole (June 2009). "Using the Census Bureau's surname list to improve estimates of race/ethnicity and associated disparities". Health Services and Outcomes Research Methodology. 9 (2): 69–83. doi:10.1007/s10742-009-0047-1. ISSN 1387-3741.
- Chen, Jiahao; Kallus, Nathan; Mao, Xiaojie; Svacha, Geoffry; Udell, Madeleine (2019). "Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved". Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* '19. Atlanta, GA, USA: ACM Press: 339–348. doi:10.1145/3287560.3287594. ISBN 9781450361255.
- Kilbertus, Niki; Gascon, Adria; Kusner, Matt; Veale, Michael; Gummadi, Krishna; Weller, Adrian (2018). "Blind Justice: Fairness with Encrypted Sensitive Attributes". International Conference on Machine Learning: 2630–2639.
- Binns, Reuben; Veale, Michael; Kleek, Max Van; Shadbolt, Nigel (13 September 2017). Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation. Social Informatics. Lecture Notes in Computer Science. 10540. pp. 405–415. arXiv:1707.01477. doi:10.1007/978-3-319-67256-4_32. ISBN 978-3-319-67255-7.
- Claburn, Thomas. "EU Data Protection Law May End The Unknowable Algorithm – InformationWeek". InformationWeek. Retrieved 25 November 2017.
- https://research.google.com/bigpicture/attacking-discrimination-in-ml/ Attacking discrimination with smarter machine learning
- Hardt, Moritz; Price, Eric; Srebro, Nathan (2016). "Equality of Opportunity in Supervised Learning". arXiv:1610.02413 [cs.LG].
- https://venturebeat.com/2018/05/25/microsoft-is-developing-a-tool-to-help-engineers-catch-bias-in-algorithms/ Microsoft is developing a tool to help engineers catch bias in algorithms
- https://qz.com/1268520/facebook-says-it-has-a-tool-to-detect-bias-in-its-artificial-intelligence/ Facebook says it has a tool to detect bias in its artificial intelligence
- open source) Pymetrics audit-ai
- https://venturebeat-com.cdn.ampproject.org/c/s/venturebeat.com/2018/05/31/pymetrics-open-sources-audit-ai-an-algorithm-bias-detection-tool/amp/ Pymetrics open-sources Audit AI, an algorithm bias detection tool
- https://github.com/dssg/aequitas open source Aequitas: Bias and Fairness Audit Toolkit
- https://dsapp.uchicago.edu/aequitas/ open-sources Audit AI, Aequitas at University of Chicago
- https://www.ibm.com/blogs/research/2018/02/mitigating-bias-ai-models/ Mitigating Bias in AI Models
- Koene, Ansgar (June 2017). "Algorithmic Bias: Addressing Growing Concerns [Leading Edge]". IEEE Technology and Society Magazine. 36 (2): 31–32. doi:10.1109/mts.2017.2697080. ISSN 0278-0097.
- "P7003 - Algorithmic Bias Considerations". standards.ieee.org. Retrieved 2018-12-03.
- Bygrave, Lee A (2001). "AUTOMATED PROFILING". Computer Law & Security Review. 17 (1): 17–24. doi:10.1016/s0267-3649(01)00104-2.
- Veale, Michael; Edwards, Lilian (2018). "Clarity, Surprises, and Further Questions in the Article 29 Working Party Draft Guidance on Automated Decision-Making and Profiling" (PDF). Computer Law & Security Review. doi:10.2139/ssrn.3071679. SSRN 3071679.
- Wachter, Sandra; Mittelstadt, Brent; Floridi, Luciano (1 May 2017). "Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation". International Data Privacy Law. 7 (2): 76–99. doi:10.1093/idpl/ipx005. ISSN 2044-3994.
- Edwards, Lilian; Veale, Michael (23 May 2017). "Slave to the Algorithm? Why a Right to an Explanation Is Probably Not the Remedy You Are Looking For". Duke Law & Technology Review. 16: 18–84. doi:10.2139/ssrn.2972855. SSRN 2972855.
- Singer, Natasha (2 February 2013). "Consumer Data Protection Laws, an Ocean Apart". The New York Times. Retrieved 26 November 2017.
- Obama, Barack (12 October 2016). "The Administration's Report on the Future of Artificial Intelligence". whitehouse.gov. National Archives. Retrieved 26 November 2017.
- and Technology Council, National Science (2016). National Artificial Intelligence Research and Development Strategic Plan (PDF). US Government. Retrieved 26 November 2017.
- Kirchner, Lauren (18 December 2017). "New York City Moves to Create Accountability for Algorithms — ProPublica". ProPublica. ProPublica. Retrieved 28 July 2018.
- "The New York City Council - File #: Int 1696-2017". legistar.council.nyc.gov. New York City Council. Retrieved 28 July 2018.
- Powles, Julia. "New York City's Bold, Flawed Attempt to Make Algorithms Accountable". The New Yorker. The New Yorker. Retrieved 28 July 2018.
- "India Weighs Comprehensive Data Privacy Bill, Similar to EU's GDPR". Insurance Journal. 2018-07-31. Retrieved 2019-02-26.
- Noble, Safiya Umoja (2018). Algorithms of oppression: How search engines reinforce racism. New York: New York University Press. ISBN 9781479837243.