Eliezer Yudkowsky

From Wikipedia, the free encyclopedia
Jump to: navigation, search
Eliezer Yudkowsky
Eliezer Yudkowsky, Stanford 2006 (square crop).jpg
Born (1979-09-11) September 11, 1979 (age 36)
Nationality American
Fields Theoretical computer science
Institutions Machine Intelligence Research Institute
Spouse Brienne Yudkowsky (m. 2013)[1]

Eliezer Shlomo Yudkowsky (born September 11, 1979) is an American writer, blogger, and advocate for friendly artificial intelligence.[2][3] He is a Research Fellow and co-founder at the Machine Intelligence Research Institute, a private research nonprofit based in Berkeley, California.[4]

Work in Artificial Intelligence safety[edit]

Goal learning and incentives in software systems[edit]

Yudkowsky's views on the safety challenges posed by future generations of AI systems are discussed in the standard undergraduate textbook in AI, Stuart Russell and Peter Norvig's Artificial Intelligence: A Modern Approach. Noting the difficulty of formally specifying general-purpose goals by hand, Russell and Norvig cite Yudkowsky's proposal that autonomous and adaptive systems be designed to learn correct behavior over time:

Yudkowsky (2008)[5] goes into more detail about how to design a Friendly AI. He asserts that friendliness (a desire not to harm humans) should be designed in from the start, but that the designers should recognize both that their own designs may be flawed, and that the robot will learn and evolve over time. Thus the challenge is one of mechanism design — to design a mechanism for evolving AI under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes.[2]

Citing Steve Omohundro's idea of instrumental convergence, Russell and Norvig caution that autonomous decision-making systems with poorly designed goals would have default incentives to treat humans adversarially, or as dispensable resources, unless specifically designed to counter such incentives: "even if you only want your program to play chess or prove theorems, if you give it the capability to learn and alter itself, you need safeguards".[2][6]

In response to the instrumental convergence concern, Yudkowsky and other MIRI researchers have recommended that work be done to specify software agents that converge on safe default behaviors even when their goals are misspecified.[7] The Future of Life Institute (FLI) summarizes this research program in the Open Letter on Artificial Intelligence research priorities document:

If an AI system is selecting the actions that best allow it to complete a given task, then avoiding conditions that prevent the system from continuing to pursue the task is a natural subgoal (and conversely, seeking unconstrained situations is sometimes a useful heuristic). This could become problematic, however, if we wish to repurpose the system, to deactivate it, or to significantly alter its decision-making process; such a system would rationally avoid these changes. Systems that do not exhibit these behaviors have been termed corrigible systems, and both theoretical and practical work in this area appears tractable and useful. For example, it may be possible to design utility functions or decision processes so that a system will not try to avoid being shut down or repurposed, and theoretical frameworks could be developed to better understand the space of potential systems that avoid undesirable behaviors.[8]

Yudkowsky argues that as AI systems become increasingly intelligent, new formal tools will be needed in order to avert default incentives for harmful behavior, as well as to inductively teach correct behavior.[7][9] These lines of research are discussed in MIRI's 2015 technical agenda.[10]

System reliability and transparency[edit]

Yudkowsky studies decision theories that achieve better outcomes than causal decision theory in Newcomblike problems.[11] This includes decision procedures that allow agents to cooperate with equivalent reasoners in the one-shot prisoner's dilemma.[12] Yudkowsky has also written on theoretical prerequisites for self-verifying software.[13][9]

Yudkowsky argues that it is important for advanced AI systems to be cleanly designed and transparent to human inspection, both to ensure stable behavior and to allow greater human oversight and analysis.[9] Citing papers on this topic by Yudkowsky and other MIRI researchers, the FLI research priorities document states that work on defining correct reasoning in embodied and logically non-omniscient agents would be valuable for the design, use, and oversight of AI agents.[8][14]

Capabilities forecasting[edit]

In their discussion of Omohundro and Yudkowsky's work, Russell and Norvig cite I. J. Good's 1965 prediction that when computer systems begin to outperform humans in software engineering tasks, this may result in a feedback loop of increasingly capable AI systems. This raises the possibility that AI's impact could increase very quickly after it reaches a certain level of capability.[2]

In the intelligence explosion scenario inspired by Good's hypothetical, recursively self-improving AI systems quickly transition from subhuman general intelligence to superintelligent.[9] Nick Bostrom's 2014 book Superintelligence: Paths, Dangers, Strategies sketches out Good's argument in greater detail, while making a broader case for expecting AI systems to eventually outperform humans across the board. Bostrom cites writing by Yudkowsky on inductive value learning and on the risk of anthropomorphizing advanced AI systems, e.g.: "AI might make an apparently sharp jump in intelligence purely as the result of anthropomorphism, the human tendency to think of 'village idiot' and 'Einstein' as the extreme ends of the intelligence scale, instead of nearly indistinguishable points on the scale of minds-in-general."[5][15]

The Open Philanthropy Project, an offshoot of the charity evaluator GiveWell, credits Yudkowsky and Bostrom with several (paraphrased) arguments for expecting future AI advances to have a large societal impact:[16]

Over a relatively short geological timescale, humans have come to have enormous impacts on the biosphere, often leaving the welfare of other species dependent on the objectives and decisions of humans. It seems plausible that the intellectual advantages humans have over other animals have been crucial in allowing humans to build up the scientific and technological capabilities that have made this possible. If advanced artificial intelligence agents become significantly more powerful than humans, it seems possible that they could become the dominant force in the biosphere, leaving humans' welfare dependent on their objectives and decisions. As with the interaction between humans and other species in the natural environment, these problems could be the result of competition for resources rather than malice.

In comparison with other evolutionary changes, there was relatively little time between our hominid ancestors and the evolution of humans. There was therefore relatively little time for evolutionary pressure to lead to improvements in human intelligence relative to the intelligence of our hominid ancestors, suggesting that the increases in intelligence may be small on some absolute scale. [...T]his makes it seem plausible that creating intelligent agents that are more intelligent than humans could have dramatic real-world consequences even if the difference in intelligence is small in an absolute sense.[14]

Russell and Norvig raise the objection that there are known limits to intelligent problem-solving from computational complexity theory; if there are strong limits on how efficiently algorithms can solve various computer science tasks, then intelligence explosion may not be possible.[2] Yudkowsky has debated the likelihood of intelligence explosion with economist Robin Hanson, who argues that AI progress is likely to accelerate over time, but is not likely to be localized or discontinuous.[17]

In a 2013 report, Intelligence Explosion Microeconomics, Yudkowsky suggests that more formalized microeconomic model of cognition and self-improvement could yield better predictions about whether intelligence explosion is feasible. In contrast, Yudkowsky expresses skepticism about our ability to predict when AI algorithms are likely to exceed humans in general intelligence, even if we attempt to improve our timelines with more formal models.[16]

Rationality writing[edit]

Between 2006 and 2009, Yudkowsky and Hanson were the principal contributors to Overcoming Bias,[18] a cognitive and social science blog sponsored by the Future of Humanity Institute of Oxford University. In February 2009, Yudkowsky founded LessWrong,[19] a "community blog devoted to refining the art of human rationality".[20] Overcoming Bias has since functioned as Hanson's personal blog. LessWrong has been covered in depth in Business Insider,[21] and core concepts from LessWrong have been referenced in columns in The Guardian.[22][23]

Yudkowsky has also written several works of fiction.[24] His fan fiction story, Harry Potter and the Methods of Rationality, uses plot elements from J.K. Rowling's Harry Potter series to illustrate topics in science.[20][25][26][27][28][29][30] The New Yorker describes Harry Potter and the Methods of Rationality as a retelling of Rowling's original "in an attempt to explain Harry’s wizardry through the scientific method".[31]

Blogposts by Yudkowsky have been released as six books, in an ebook titled Rationality: From AI to Zombies in March 2015 by the Machine Intelligence Research Institute, which is also planning to release the books as printed versions.[32]

Academic publications[edit]

See also[edit]


  1. ^ Yudkowsky, Eliezer. "Eliezer S. Yudkowsky". yudkowsky.net. Retrieved 7 October 2015. 
  2. ^ a b c d e Russell, Stuart; Norvig, Peter (2009). Artificial Intelligence: A Modern Approach. Prentice Hall. ISBN 978-0-13-604259-4. 
  3. ^ Leighton, Jonathan (2011). The Battle for Compassion: Ethics in an Apathetic Universe. Algora. ISBN 978-0-87586-870-7. 
  4. ^ Kurzweil, Ray (2005). The Singularity Is Near. New York, US: Viking Penguin. ISBN 0-670-03384-7. 
  5. ^ a b Yudkowsky, Eliezer (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (PDF). In Bostrom, Nick; Ćirković, Milan. Global Catastrophic Risks. Oxford University Press. ISBN 978-0199606504. 
  6. ^ Omohundro, Steve (2008). "The Basic AI Drives" (PDF). Proceedings of the First AGI Conference. IOS Press. 
  7. ^ a b Soares, Nate; Fallenstein, Benja; Yudkowsky, Eliezer (2015). "Corrigibility". AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, January 25–26, 2015. AAAI Publications. 
  8. ^ a b Future of Life Institute (2015). Research priorities for robust and beneficial artificial intelligence (PDF) (Report). Retrieved 12 October 2015. 
  9. ^ a b c d Yudkowsky, Eliezer (2013). "Five theses, two lemmas, and a couple of strategic implications". MIRI Blog. Retrieved 12 October 2015. 
  10. ^ Soares, Nate; Fallenstein, Benja (2015). "Aligning Superintelligence with Human Interests: A Technical Research Agenda" (PDF). In Miller, James; Yampolskiy, Roman; Armstrong, Stuart; et al. The Technological Singularity: Managing the Journey. Springer. 
  11. ^ Soares, Nate; Fallenstein, Benja (2015). "Toward Idealized Decision Theory". arXiv:1507.01986 [cs.AI]. 
  12. ^ LaVictoire, Patrick; Fallenstein, Benja; Yudkowsky, Eliezer; Bárász, Mihály; Christiano, Paul; Herreshoff, Marcello (2014). "Program Equilibrium in the Prisoner’s Dilemma via Löb’s Theorem". Multiagent Interaction without Prior Coordination: Papers from the AAAI-14 Workshop. AAAI Publications. 
  13. ^ Fallenstein, Benja; Soares, Nate (2015). Vingean Reflection: Reliable Reasoning for Self-Improving Agents (PDF) (Technical report). Machine Intelligence Research Institute. 2015-2. 
  14. ^ a b GiveWell (2015). Potential risks from advanced artificial intelligence (Report). Retrieved 12 October 2015. 
  15. ^ Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. ISBN 0199678111. 
  16. ^ a b Yudkowsky, Eliezer (2013). Intelligence Explosion Microeconomics (PDF) (Technical report). Machine Intelligence Research Institute. 2013-1. 
  17. ^ Hanson, Robin; Yudkowsky, Eliezer (2013). The Hanson-Yudkowsky AI Foom Debate. Machine Intelligence Research Institute. 
  18. ^ "Overcoming Bias: About". Robin Hanson. Retrieved 1 February 2012. 
  19. ^ "Where did Less Wrong come from? (LessWrong FAQ)". Retrieved 11 September 2014. 
  20. ^ a b Miller, James (2012). Singularity Rising. ISBN 978-1936661657. 
  21. ^ Miller, James (July 28, 2011). "You Can Learn How To Become More Rational". Business Insider. Retrieved 25 March 2014. 
  22. ^ Burkeman, Oliver (July 8, 2011). "This column will change your life: Feel the ugh and do it anyway. Can the psychological flinch mechanism be beaten?". The Guardian. Retrieved 25 March 2014. 
  23. ^ Burkeman, Oliver (March 9, 2012). "This column will change your life: asked a tricky question? Answer an easier one. We all do it, all the time. So how can we get rid of this eccentricity?". The Guardian. Retrieved March 25, 2014. 
  24. ^ Eliezer S. Yudkowsky. "Fiction". Yudkowsky. Retrieved 14 September 2015. 
  25. ^ David Brin (2010-06-21). "CONTRARY BRIN: A secret of college life... plus controversies and science!". Davidbrin.blogspot.com. Retrieved 31 August 2012. "'Harry Potter' and the Key to Immortality", Daniel Snyder, The Atlantic
  26. ^ Authors (2012-04-02). "Rachel Aaron interview (April 2012)". Fantasybookreview.co.uk. Retrieved 31 August 2012. 
  27. ^ "Civilian Reader: An Interview with Rachel Aaron". Civilian-reader.blogspot.com. 2011-05-04. Retrieved 31 August 2012. 
  28. ^ Hanson, Robin (2010-10-31). "Hyper-Rational Harry". Overcoming Bias. Retrieved 31 August 2012. 
  29. ^ Swartz, Aaron. "The 2011 Review of Books (Aaron Swartz's Raw Thought)". archive.org. Archived from the original on 2013-03-16. Retrieved 4 October 2013. 
  30. ^ "Harry Potter and the Methods of Rationality". fanfiction.net. 2010-02-28. Retrieved 29 December 2014. 
  31. ^ Packer, George (2011). "No Death, No Taxes: The Libertarian Futurism of a Silicon Valley Billionaire". The New Yorker: 54. Retrieved 12 October 2015. 
  32. ^ [1]

External links[edit]