Machine Intelligence Research Institute: Difference between revisions
knock yourself out. leaving the tag; please remove it when you are done. |
|||
Line 71: | Line 71: | ||
===Forecasting=== |
===Forecasting=== |
||
MIRI studies questions regarding what we can or cannot predict about future AI technology, how we can improve our forecasting ability, and which interventions available today appear to be the most beneficial.<ref>{{cite web |last1=Gallagher |first1=Brian |title=Scary AI Is More “Fantasia” Than “Terminator” |url=http://nautil.us/issue/58/self/scary-ai-is-more-fantasia-than-terminator |website=Nautilus |accessdate=28 July 2018}}</ref> |
MIRI studies questions regarding what we can or cannot predict about future AI technology, how we can improve our forecasting ability, and which interventions available today appear to be the most beneficial.<ref>{{cite book |last1=Bostrom |first1=Nick |author-link1=Nick Bostrom |last2=Yudkowsky |first2=Eliezer |author-link2=Eliezer Yudkowsky |date=2014 |chapter=The Ethics of Artificial Intelligence |chapterurl=https://intelligence.org/files/EthicsofAI.pdf |editor1-last=Frankish |editor1-first=Keith |editor2-last=Ramsey |editor2-first=William |title=The Cambridge Handbook of Artificial Intelligence |url= |location=New York |publisher=Cambridge University Press |page= |isbn=978-0-521-87142-6}}</ref> Soares believes that as AI becomes more advanced, it will be more difficult to understand its workings and impacts.<ref>{{cite web |last1=Gallagher |first1=Brian |title=Scary AI Is More “Fantasia” Than “Terminator” |url=http://nautil.us/issue/58/self/scary-ai-is-more-fantasia-than-terminator |website=Nautilus |accessdate=28 July 2018}}</ref> |
||
Beginning in 2014, MIRI has funded forecasting work through the independent AI Impacts project. AI Impacts studies historical instances of discontinuous technological change, and has developed new measures of the relative computational power of humans and computer hardware.<ref>{{cite news |last=Hsu |first=Jeremy |date=2015 |title=Making Sure AI's Rapid Rise Is No Surprise |url=http://blogs.discovermagazine.com/lovesick-cyborg/2015/09/02/making-sure-ais-rapid-rise-is-no-surprise/ |newspaper=[[Discover (magazine)|Discover]] |location= |access-date=12 October 2015}}</ref><ref>{{cite news |last=De Looper |first=Christian |date=2015 |title=Research Suggests Human Brain Is 30 Times As Powerful As The Best Supercomputers |url=http://www.techtimes.com/articles/79701/20150826/research-suggusts-human-brain-30-times-powerful-best-supercomputers.htm |newspaper=Tech Times |location= |access-date=12 October 2015}}</ref> |
|||
MIRI researchers' interest in discontinuous AI progress stems from [[I. J. Good]]'s argument that sufficiently advanced AI systems will eventually outperform humans in software engineering tasks, leading to a feedback loop of increasingly capable AI systems:<ref name="tech"/><ref name="gcr">{{cite book |last=Yudkowsky |first=Eliezer |author-link=Eliezer Yudkowsky |date=2008 |chapter=Artificial Intelligence as a Positive and Negative Factor in Global Risk |chapterurl=https://intelligence.org/files/AIPosNegFactor.pdf |editor1-last=Bostrom |editor1-first=Nick |editor1-link=Nick Bostrom |editor2-last=Ćirković |editor2-first=Milan |title=Global Catastrophic Risks |url= |location= |publisher=Oxford University Press |page= |isbn=978-0199606504}}</ref><ref name="priorities"/><ref name="atlantic">{{cite news |last=LaFrance |first=Adrienne |date=2015 |title=Building Robots With Better Morals Than Humans |url=https://www.theatlantic.com/technology/archive/2015/01/building-robots-with-better-morals-than-humans/385015/ |newspaper=[[The Atlantic]] |location= |access-date=12 October 2015}}</ref> |
|||
Bostrom and researchers at MIRI have expressed skepticism about the views of singularity advocates like [[Ray Kurzweil]] that superintelligence is "just around the corner". MIRI researchers have advocated early safety work as a precautionary measure, while arguing that predictions of AI progress have not proven reliable.<ref name="atlantic"/><ref name="superintelligence"/> |
|||
<blockquote>Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.<ref>{{cite journal |last=Good |first=Irving |author-link=I. J. Good |date=1965 |title=Speculations Concerning the First Ultraintelligent Machine |url=http://www.kushima.org/is/wp-content/uploads/2015/07/Good65ultraintelligent.pdf |journal=Advances in Computers |publisher= |volume=6 |issue= |pages= |doi= |access-date=4 October 2015}}</ref></blockquote> |
|||
Eliezer Yudkowsky, MIRI's co-founder and senior researcher, writes on the importance of [[friendly artificial intelligence]] in smarter-than-human systems.<ref name="aima">{{cite book |last1=Russell |first1=Stuart |author1-link=Stuart J. Russell |last2=Norvig |first2=Peter |author2-link=Peter Norvig |date=2009 |title=[[Artificial Intelligence: A Modern Approach]] |url= |location= |publisher=Prentice Hall |page= |isbn=978-0-13-604259-4}}</ref> |
|||
Writers like Bostrom use the term ''[[superintelligence]]'' in place of Good's ''ultraintelligence''.<ref name="superintelligence">{{cite book|last1=Bostrom|first1=Nick|author-link=Nick Bostrom|title=[[Superintelligence: Paths, Dangers, Strategies]]|date=2014|isbn=0199678111|edition=First edition.|quote=}}</ref> Following [[Vernor Vinge]], Good's idea of [[intelligence explosion]] has come to be associated with the idea of a "[[technological singularity]]".<ref>{{cite journal |last=Vinge |first=Vernor |author-link=Vernor Vinge |date=1993 |title=The Coming Technological Singularity: How to Survive in the Post-Human Era |url=https://www-rohan.sdsu.edu/faculty/vinge/misc/singularity.html |journal=[[Whole Earth Review]] |location= |publisher= |access-date=12 October 2015}}</ref><ref name="aima">{{cite book |last1=Russell |first1=Stuart |author1-link=Stuart J. Russell |last2=Norvig |first2=Peter |author2-link=Peter Norvig |date=2009 |title=[[Artificial Intelligence: A Modern Approach]] |url= |location= |publisher=Prentice Hall |page= |isbn=978-0-13-604259-4}}</ref><ref>{{cite web |url=https://intelligence.org/2007/09/30/three-major-singularity-schools/ |title=Three Major Singularity Schools |last=Yudkowsky |first=Eliezer |author-link=Eliezer Yudkowsky |date=2007 |website=MIRI Blog |publisher= |access-date=11 October 2015 |quote=}}</ref> Bostrom and researchers at MIRI have expressed skepticism about the views of singularity advocates like [[Ray Kurzweil]] that superintelligence is "just around the corner". MIRI researchers have advocated early safety work as a precautionary measure, while arguing that predictions of AI progress have not proven reliable.<ref>{{cite web |url=https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/ |title=Brooks and Searle on AGI volition and timelines |last=Bensinger |first=Rob |date=2015 |website=MIRI Blog |publisher= |access-date=12 October 2015 |quote=}}</ref><ref name="atlantic"/><ref name="superintelligence"/> |
|||
Eliezer Yudkowsky, MIRI's co-founder and senior researcher, is frequently cited for his writing on the long-term social impact of progress in AI. Russell and Norvig's ''[[Artificial Intelligence: A Modern Approach]]'', A textbook in the field of AI, summarizes Yudkowsky's thesis: |
|||
<blockquote>If ultraintelligent machines are a possibility, we humans would do well to make sure that we design their predecessors in such a way that they design themselves to treat us well. [...] Yudkowsky (2008)<ref name="gcr"/> goes into more detail about how to design a ''Friendly AI''. He asserts that friendliness (a desire not to harm humans) should be designed in from the start, but that the designers should recognize both that their own designs may be flawed, and that the robot will learn and evolve over time. Thus the problem is one of mechanism design—to define a mechanism for evolving AI systems under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes.<ref name="aima"/></blockquote> |
|||
Yudkowsky writes on the importance of [[friendly artificial intelligence]] in smarter-than-human systems.<ref>{{cite book|last1=Tegmark|first1=Max|author-link=Max Tegmark|title=[[Our Mathematical Universe: My Quest for the Ultimate Nature of Reality]]|date=2014|isbn=9780307744258|edition=First edition.|chapter=Life, Our Universe and Everything|quote=}}</ref> This informal goal is reflected in MIRI's recent publications as the requirement that AI systems be "aligned with human interests".<ref name="tech"/> Following Bostrom and [[Steve Omohundro]], MIRI researchers believe that autonomous [[artificial general intelligence|generally intelligent]] AI systems will have default incentives to treat human operators as competitors, obstacles, or threats if they are not specifically designed to promote their operators' goals.<ref name="corrigibility">{{cite conference |
|||
|url=http://aaai.org/ocs/index.php/WS/AAAIW15/paper/view/10124/10136 |
|||
|title=Corrigibility |
|||
|last1=Soares |
|||
|first1=Nate |
|||
|last2=Fallenstein |
|||
|first2=Benja |
|||
|last3=Yudkowsky |
|||
|first3=Eliezer |
|||
|author-link3=Eliezer Yudkowsky |
|||
|last4=Armstrong |
|||
|first4=Stuart |
|||
|date=2015 |
|||
|publisher=AAAI Publications |
|||
|book-title=AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, January 25–26, 2015 |pages= |location= |id= }}</ref><ref>{{cite conference |
|||
|url=https://intelligence.org/files/ComplexValues.pdf |
|||
|title=The Basic AI Drives |
|||
|last=Omohundro |
|||
|first=Steve |
|||
|author-link=Steve Omohundro |
|||
|date=2008 |
|||
|publisher=IOS |
|||
|book-title=Artificial General Intelligence 2008: Proceedings of the First AGI Conference |
|||
|format=PDF |
|||
|location=Amsterdam}}</ref><ref name="superintelligence"/><ref name="post"/> |
|||
===High reliability and error tolerance in AI=== |
|||
The Future of Life Institute (FLI) research priorities document states: |
|||
<blockquote>Mathematical tools such as formal logic, probability, and decision theory have yielded significant insight into the foundations of reasoning and decision-making. However, there are still many open problems in the foundations of reasoning and decision. Solutions to these problems may make the behavior of very capable systems much more reliable and predictable. Example research topics in this area include reasoning and decision under bounded computational resources à la Horvitz and Russell, how to take into account correlations between AI systems' behaviors and those of their environments or of other agents, how agents that are embedded in their environments should reason, and how to reason about uncertainty over logical consequences of beliefs or other deterministic computations. These topics may benefit from being considered together, since they appear deeply linked.<ref name="priorities"/></blockquote> |
|||
The priorities document cites MIRI publications in the relevant areas: formalizing cooperation in the [[prisoner's dilemma]] between "[[superrationality|superrational]]" software agents;<ref>{{cite conference |url=http://www.aaai.org/ocs/index.php/WS/AAAIW14/paper/viewFile/8833/8294 |title=Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem |last1=LaVictoire |first1=Patrick |last2=Fallenstein |first2=Benja |last3=Yudkowsky |first3=Eliezer |author-link3=Eliezer Yudkowsky |last4=Bárász |first4=Mihály |last5=Christiano |first5=Paul |last6=Herreshoff |first6=Marcello |date=2014 |publisher=AAAI Publications |book-title=Multiagent Interaction without Prior Coordination: Papers from the AAAI-14 Workshop |pages= |location= |id= }}</ref> defining alternatives to [[causal decision theory]] and [[evidential decision theory]] in [[Newcomb's problem]];<ref name="decision">{{cite arXiv |last1=Soares |first1=Nate |last2=Fallenstein |first2=Benja |eprint=1507.01986 |title=Toward Idealized Decision Theory |class=cs.AI |date=2015}}</ref> and developing alternatives to [[Solomonoff's theory of inductive inference]] for agents embedded in physical environments<ref>{{cite techreport |last=Soares |first=Nate |title=Formalizing Two Problems of Realistic World-Models |number=2015-3 |institution=Machine Intelligence Research Institute |url=https://intelligence.org/files/RealisticWorldModels.pdf |date= |year=2015}}</ref> and agents reasoning without logical omniscience.<ref>{{cite techreport |last1=Soares |first1=Nate |last2=Fallenstein |first2=Benja |title=Questions of Reasoning under Logical Uncertainty |number=2015-1 |institution=Machine Intelligence Research Institute |url=https://intelligence.org/files/QuestionsLogicalUncertainty.pdf |date= |year=2015}}</ref><ref name="atlantic"/> |
|||
Standard decision procedures are not well-specified enough (e.g., with regard to counterfactuals) to be instantiated as algorithms. MIRI researcher Benja Fallenstein and then-researcher Nate Soares write that causal decision theory is "unstable under reflection" in the sense that a rational agent following causal decision theory "correctly identifies that the agent should modify itself to stop using CDT [causal decision theory] to make decisions". MIRI researchers identify "logical decision theories" as alternatives that perform better in general decision-making tasks.<ref name="decision"/> |
|||
MIRI also studies self-monitoring and self-verifying software. The FLI research priorities document notes that "a formal system that is sufficiently powerful cannot use formal methods in the obvious way to gain assurance about the accuracy of functionally similar formal systems, on pain of inconsistency via [[Gödel's incompleteness theorems]]".<ref name="priorities"/> MIRI's publications on ''Vingean reflection'' attempt to model the Gödelian limits on self-referential reasoning and identify practically useful exceptions.<ref>{{cite techreport |last2=Soares |first2=Nate |last1=Fallenstein |first1=Benja |title=Vingean Reflection: Reliable Reasoning for Self-Improving Agents |number=2015-2 |institution=Machine Intelligence Research Institute |url=https://intelligence.org/files/VingeanReflection.pdf |date= |year=2015}}</ref> |
|||
Soares and Fallenstein classify the above research programs as aimed at high reliability and transparency in agent behavior. They separately recommend research into "error-tolerant" software systems, citing [[human error]] and default incentives as sources of serious risk.<ref name="corrigibility"/><ref name="post"/> The FLI research priorities document adds: |
|||
<blockquote>If an AI system is selecting the actions that best allow it to complete a given task, then avoiding conditions that prevent the system from continuing to pursue the task is a natural subgoal (and conversely, seeking unconstrained situations is sometimes a useful heuristic). This could become problematic, however, if we wish to repurpose the system, to deactivate it, or to significantly alter its decision-making process; such a system would rationally avoid these changes. Systems that do not exhibit these behaviors have been termed ''corrigible'' systems, and both theoretical and practical work in this area appears tractable and useful.</blockquote> |
|||
Five researchers at MIRI produced a paper called ''Logical Induction''<ref>{{cite web |title=Workshop in Economic Theory (Joint with Applied Theory Workshop) |url=https://economics.uchicago.edu/content/workshop-economic-theory-joint-applied-theory-workshop |website=economics.uchicago.edu |publisher=University of Chicago |accessdate=28 July 2018}}</ref> on the manner in which an agent might assign probabilities to logical statements. |
|||
MIRI's priorities in these areas are summarized in their 2015 technical agenda.<ref name="tech"/> |
|||
===Value specification=== |
===Value specification=== |
Revision as of 18:07, 28 July 2018
![]() | This article is actively undergoing a major edit for a little while. To help avoid edit conflicts, please do not edit this page while this message is displayed. This page was last edited at 18:07, 28 July 2018 (UTC) (6 years ago) – this estimate is cached, . Please remove this template if this page hasn't been edited for a significant time. If you are the editor who added this template, please be sure to remove it or replace it with {{Under construction}} between editing sessions. |
![]() | This article contains content that is written like an advertisement. (July 2018) |
File:MIRI logo.png | |
Formation | 2000[1] |
---|---|
Type | Nonprofit research institute |
Legal status | 501(c)(3) tax exempt charity |
Purpose | Research into friendly artificial intelligence |
Location | |
Edwin Evans | |
Nate Soares | |
Key people | Eliezer Yudkowsky |
Revenue | $1.7 million (2013)[2] |
Staff | 14[3] |
Website | intelligence |
Formerly called | Singularity Institute, Singularity Institute for Artificial Intelligence |
The Machine Intelligence Research Institute (MIRI), formerly the Singularity Institute for Artificial Intelligence (SIAI), is a non-profit organization founded in 2000[1] to research safety issues related to the development of advanced artificial intelligence.
MIRI aims to create new tools and protocol to ensure the safety of future AI technology.[4] The organization hosts regular research workshops to develop mathematical foundations for this project,[5] and has been cited as one of several academic and nonprofit groups studying long-term AI outcomes.[6][7][8]
History
In 2000, Eliezer Yudkowsky, Brian and Sabine Atkins founded the Singularity Institute for Artificial Intelligence to "help humanity prepare for the moment when machine intelligence exceeded human intelligence".[9][10][11] In early 2005, SIAI relocated from Atlanta, Georgia to Silicon Valley. From 2006 to 2012, the Institute collaborated with Singularity University to produce the Singularity Summit, a science and technology conference. Speakers included Steven Pinker, Peter Norvig, Stephen Wolfram, John Tooby, James Randi, and Douglas Hofstadter.[12][13][14].
In mid-2012, the Institute spun off a new organization called the Center for Applied Rationality, whose focus is on using ideas from cognitive science to improve people's effectiveness in their daily lives.[15][16][17] Having previously shortened its name to "Singularity Institute", in January 2013 SIAI changed its name to the "Machine Intelligence Research Institute" in order to avoid confusion with Singularity University. MIRI gave control of the Singularity Summit to Singularity University and shifted its focus toward research in mathematics and theoretical computer science.[18]
In mid-2014, Nick Bostrom's book Superintelligence: Paths, Dangers, Strategies helped spark public discussion about the risk of unsafe artificial general intelligence, receiving endorsements from Bill Gates and Elon Musk.[19][20][21][22] Stephen Hawking and AI pioneer Stuart Russell co-authored a Huffington Post article citing the work of MIRI and other organizations in the area:
Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all. [...] Although we are facing potentially the best or worst thing ever to happen to humanity, little serious research is devoted to these issues outside small non-profit institutes such as the Cambridge Center for Existential Risk, the Future of Humanity Institute, the Machine Intelligence Research Institute, and the Future of Life Institute.[7]
In early 2015, MIRI's research was cited in a research priorities document accompanying an open letter on AI that called for "expanded research aimed at ensuring that increasingly capable AI systems are robust and beneficial".[23] Musk responded by funding a large AI safety grant program, with grant recipients including Bostrom, Russell, Bart Selman, Francesca Rossi, Thomas Dietterich, Manuela M. Veloso, and researchers at MIRI.[8][24] MIRI expanded as part of a general wave of increased interest in safety among other researchers in the AI community.[25]
Research
![](http://upload.wikimedia.org/wikipedia/commons/thumb/4/44/Nate_Soares_giving_a_talk_at_Google.gk.jpg/220px-Nate_Soares_giving_a_talk_at_Google.gk.jpg)
Forecasting
MIRI studies questions regarding what we can or cannot predict about future AI technology, how we can improve our forecasting ability, and which interventions available today appear to be the most beneficial.[26] Soares believes that as AI becomes more advanced, it will be more difficult to understand its workings and impacts.[27]
Beginning in 2014, MIRI has funded forecasting work through the independent AI Impacts project. AI Impacts studies historical instances of discontinuous technological change, and has developed new measures of the relative computational power of humans and computer hardware.[28][29]
MIRI researchers' interest in discontinuous AI progress stems from I. J. Good's argument that sufficiently advanced AI systems will eventually outperform humans in software engineering tasks, leading to a feedback loop of increasingly capable AI systems:[4][30][23][22]
Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.[31]
Writers like Bostrom use the term superintelligence in place of Good's ultraintelligence.[19] Following Vernor Vinge, Good's idea of intelligence explosion has come to be associated with the idea of a "technological singularity".[32][33][34] Bostrom and researchers at MIRI have expressed skepticism about the views of singularity advocates like Ray Kurzweil that superintelligence is "just around the corner". MIRI researchers have advocated early safety work as a precautionary measure, while arguing that predictions of AI progress have not proven reliable.[35][22][19]
Eliezer Yudkowsky, MIRI's co-founder and senior researcher, is frequently cited for his writing on the long-term social impact of progress in AI. Russell and Norvig's Artificial Intelligence: A Modern Approach, A textbook in the field of AI, summarizes Yudkowsky's thesis:
If ultraintelligent machines are a possibility, we humans would do well to make sure that we design their predecessors in such a way that they design themselves to treat us well. [...] Yudkowsky (2008)[30] goes into more detail about how to design a Friendly AI. He asserts that friendliness (a desire not to harm humans) should be designed in from the start, but that the designers should recognize both that their own designs may be flawed, and that the robot will learn and evolve over time. Thus the problem is one of mechanism design—to define a mechanism for evolving AI systems under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes.[33]
Yudkowsky writes on the importance of friendly artificial intelligence in smarter-than-human systems.[36] This informal goal is reflected in MIRI's recent publications as the requirement that AI systems be "aligned with human interests".[4] Following Bostrom and Steve Omohundro, MIRI researchers believe that autonomous generally intelligent AI systems will have default incentives to treat human operators as competitors, obstacles, or threats if they are not specifically designed to promote their operators' goals.[37][38][19][8]
High reliability and error tolerance in AI
The Future of Life Institute (FLI) research priorities document states:
Mathematical tools such as formal logic, probability, and decision theory have yielded significant insight into the foundations of reasoning and decision-making. However, there are still many open problems in the foundations of reasoning and decision. Solutions to these problems may make the behavior of very capable systems much more reliable and predictable. Example research topics in this area include reasoning and decision under bounded computational resources à la Horvitz and Russell, how to take into account correlations between AI systems' behaviors and those of their environments or of other agents, how agents that are embedded in their environments should reason, and how to reason about uncertainty over logical consequences of beliefs or other deterministic computations. These topics may benefit from being considered together, since they appear deeply linked.[23]
The priorities document cites MIRI publications in the relevant areas: formalizing cooperation in the prisoner's dilemma between "superrational" software agents;[39] defining alternatives to causal decision theory and evidential decision theory in Newcomb's problem;[40] and developing alternatives to Solomonoff's theory of inductive inference for agents embedded in physical environments[41] and agents reasoning without logical omniscience.[42][22]
Standard decision procedures are not well-specified enough (e.g., with regard to counterfactuals) to be instantiated as algorithms. MIRI researcher Benja Fallenstein and then-researcher Nate Soares write that causal decision theory is "unstable under reflection" in the sense that a rational agent following causal decision theory "correctly identifies that the agent should modify itself to stop using CDT [causal decision theory] to make decisions". MIRI researchers identify "logical decision theories" as alternatives that perform better in general decision-making tasks.[40]
MIRI also studies self-monitoring and self-verifying software. The FLI research priorities document notes that "a formal system that is sufficiently powerful cannot use formal methods in the obvious way to gain assurance about the accuracy of functionally similar formal systems, on pain of inconsistency via Gödel's incompleteness theorems".[23] MIRI's publications on Vingean reflection attempt to model the Gödelian limits on self-referential reasoning and identify practically useful exceptions.[43]
Soares and Fallenstein classify the above research programs as aimed at high reliability and transparency in agent behavior. They separately recommend research into "error-tolerant" software systems, citing human error and default incentives as sources of serious risk.[37][8] The FLI research priorities document adds:
If an AI system is selecting the actions that best allow it to complete a given task, then avoiding conditions that prevent the system from continuing to pursue the task is a natural subgoal (and conversely, seeking unconstrained situations is sometimes a useful heuristic). This could become problematic, however, if we wish to repurpose the system, to deactivate it, or to significantly alter its decision-making process; such a system would rationally avoid these changes. Systems that do not exhibit these behaviors have been termed corrigible systems, and both theoretical and practical work in this area appears tractable and useful.
Five researchers at MIRI produced a paper called Logical Induction[44] on the manner in which an agent might assign probabilities to logical statements.
MIRI's priorities in these areas are summarized in their 2015 technical agenda.[4]
Value specification
In defining correct goals for autonomous systems, Soares and Fallenstein write, "the 'intentions' of the operators are a complex, vague, fuzzy, context-dependent notion (Yudkowsky 2011).[45] Concretely writing out the full intentions of the operators in a machine-readable format is implausible if not impossible, even for simple tasks." Soares and Fallenstein propose that autonomous AI systems instead be designed to inductively learn the values of humans from observational data.[4]
Soares discusses several technical obstacles to value learning in AI: changes in the agent's beliefs may result in a mismatch between the agent's values and its ontology; agents that are well-behaved in training data may induct incorrect values in new domains; and human operators' moral uncertainty may make it difficult to identify or anticipate incorrect inductions.[22][46] Bostrom's Superintelligence discusses the philosophical problems raised by value learning at greater length.[19]
See also
References
- ^ a b "Transparency and Financials". Machine Intelligence Research Institute. Retrieved February 19, 2017.
- ^ "IRS Form 990" (PDF). Machine Intelligence Research Institute. 2013. Retrieved 12 October 2015.
- ^ "Team". Machine Intelligence Research Institute. 2016. Retrieved 4 October 2016.
- ^ a b c d e Soares, Nate; Fallenstein, Benja (2015). "Aligning Superintelligence with Human Interests: A Technical Research Agenda". In Miller, James; Yampolskiy, Roman; Armstrong, Stuart; et al. (eds.). The Technological Singularity: Managing the Journey. Springer.
{{cite book}}
: External link in
(help); Unknown parameter|chapterurl=
|chapterurl=
ignored (|chapter-url=
suggested) (help) - ^ "Research Workshops". Machine Intelligence Research Institute. 2013. Retrieved 11 October 2015.
- ^ GiveWell (2015). Potential risks from advanced artificial intelligence (Report). Retrieved 11 October 2015.
{{cite report}}
: Cite has empty unknown parameter:|coauthors=
(help) - ^ a b Hawking, Stephen; Tegmark, Max; Russell, Stuart; Wilczek, Frank (2014). "Transcending Complacency on Superintelligent Machines". The Huffington Post. Retrieved 11 October 2015.
- ^ a b c d Basulto, Dominic (2015). "The very best ideas for preventing artificial intelligence from wrecking the planet". The Washington Post. Retrieved 11 October 2015.
- ^ Ackerman, Elise (2008). "Annual A.I. conference to be held this Saturday in San Jose". San Jose Mercury News. Retrieved 11 October 2015.
- ^ "Singularity Institute Strategic Plan" (PDF). Machine Intelligence Research Institute. 2011. Retrieved 12 October 2015.
- ^ "Scientists Fear Day Computers Become Smarter Than Humans". Fox News Channel. Associated Press. 2007. Retrieved 12 October 2015.
- ^ Abate, Tom (2006). "Smarter than thou?". San Francisco Chronicle. Retrieved 12 October 2015.
- ^ Abate, Tom (2007). "Public meeting will re-examine future of artificial intelligence". San Francisco Chronicle. Retrieved 12 October 2015.
- ^ "Singularity Summit: An Annual Conference on Science, Technology, and the Future". Machine Intelligence Research Institute. 2012. Retrieved 12 October 2015.
- ^ Muehlhauser, Luke (2012). "July 2012 Newsletter". MIRI Blog. Retrieved 12 October 2015.
- ^ Stiefel, Todd; Metskas, Amanda K. (22 May 2013). "Julia Galef" (podcast). The Humanist Hour. Episode 083. The Humanist. Retrieved 3 March 2015.
- ^ Chen, Angela (2014). "More Rational Resolutions". The Wall Street Journal. Retrieved 5 March 2015.
- ^ Muehlhauser, Luke (2013). "We are now the "Machine Intelligence Research Institute" (MIRI)". MIRI Blog. Retrieved 12 October 2015.
- ^ a b c d e Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies (First edition. ed.). ISBN 0199678111.
- ^ Muehlhauser, Luke (2015). "Musk and Gates on superintelligence and fast takeoff". Luke Muehlhauser Blog. Retrieved 12 October 2015.
- ^ D'Orazio, Dante (2014). "Elon Musk says artificial intelligence is 'potentially more dangerous than nukes'". The Verge. Retrieved 5 October 2015.
- ^ a b c d e LaFrance, Adrienne (2015). "Building Robots With Better Morals Than Humans". The Atlantic. Retrieved 12 October 2015.
- ^ a b c d Future of Life Institute (2015). Research priorities for robust and beneficial artificial intelligence (PDF) (Report). Retrieved 4 October 2015.
{{cite report}}
: Cite has empty unknown parameter:|coauthors=
(help) - ^ "2015 Awardees". Future of Life Institute. 2015. Retrieved 5 October 2015.
- ^ Tegmark, Max (2017). Life 3.0: Being Human in the Age of Artificial Intelligence. United States: Knopf. ISBN 978-1-101-94659-6.
{{cite book}}
:|access-date=
requires|url=
(help) - ^ Bostrom, Nick; Yudkowsky, Eliezer (2014). "The Ethics of Artificial Intelligence". In Frankish, Keith; Ramsey, William (eds.). The Cambridge Handbook of Artificial Intelligence. New York: Cambridge University Press. ISBN 978-0-521-87142-6.
{{cite book}}
: External link in
(help); Unknown parameter|chapterurl=
|chapterurl=
ignored (|chapter-url=
suggested) (help) - ^ Gallagher, Brian. "Scary AI Is More "Fantasia" Than "Terminator"". Nautilus. Retrieved 28 July 2018.
- ^ Hsu, Jeremy (2015). "Making Sure AI's Rapid Rise Is No Surprise". Discover. Retrieved 12 October 2015.
- ^ De Looper, Christian (2015). "Research Suggests Human Brain Is 30 Times As Powerful As The Best Supercomputers". Tech Times. Retrieved 12 October 2015.
- ^ a b Yudkowsky, Eliezer (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk". In Bostrom, Nick; Ćirković, Milan (eds.). Global Catastrophic Risks. Oxford University Press. ISBN 978-0199606504.
{{cite book}}
: External link in
(help); Unknown parameter|chapterurl=
|chapterurl=
ignored (|chapter-url=
suggested) (help) - ^ Good, Irving (1965). "Speculations Concerning the First Ultraintelligent Machine" (PDF). Advances in Computers. 6. Retrieved 4 October 2015.
- ^ Vinge, Vernor (1993). "The Coming Technological Singularity: How to Survive in the Post-Human Era". Whole Earth Review. Retrieved 12 October 2015.
- ^ a b Russell, Stuart; Norvig, Peter (2009). Artificial Intelligence: A Modern Approach. Prentice Hall. ISBN 978-0-13-604259-4.
- ^ Yudkowsky, Eliezer (2007). "Three Major Singularity Schools". MIRI Blog. Retrieved 11 October 2015.
- ^ Bensinger, Rob (2015). "Brooks and Searle on AGI volition and timelines". MIRI Blog. Retrieved 12 October 2015.
- ^ Tegmark, Max (2014). "Life, Our Universe and Everything". Our Mathematical Universe: My Quest for the Ultimate Nature of Reality (First edition. ed.). ISBN 9780307744258.
- ^ a b Soares, Nate; Fallenstein, Benja; Yudkowsky, Eliezer; Armstrong, Stuart (2015). "Corrigibility". AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, January 25–26, 2015. AAAI Publications.
- ^ Omohundro, Steve (2008). "The Basic AI Drives" (PDF). Artificial General Intelligence 2008: Proceedings of the First AGI Conference. Amsterdam: IOS.
- ^ LaVictoire, Patrick; Fallenstein, Benja; Yudkowsky, Eliezer; Bárász, Mihály; Christiano, Paul; Herreshoff, Marcello (2014). "Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem". Multiagent Interaction without Prior Coordination: Papers from the AAAI-14 Workshop. AAAI Publications.
- ^ a b Soares, Nate; Fallenstein, Benja (2015). "Toward Idealized Decision Theory". arXiv:1507.01986 [cs.AI].
- ^ Soares, Nate (2015). Formalizing Two Problems of Realistic World-Models (PDF) (Technical report). Machine Intelligence Research Institute. 2015-3.
- ^ Soares, Nate; Fallenstein, Benja (2015). Questions of Reasoning under Logical Uncertainty (PDF) (Technical report). Machine Intelligence Research Institute. 2015-1.
- ^ Fallenstein, Benja; Soares, Nate (2015). Vingean Reflection: Reliable Reasoning for Self-Improving Agents (PDF) (Technical report). Machine Intelligence Research Institute. 2015-2.
- ^ "Workshop in Economic Theory (Joint with Applied Theory Workshop)". economics.uchicago.edu. University of Chicago. Retrieved 28 July 2018.
- ^ Yudkowsky, Eliezer (2011). "Complex Value Systems in Friendly AI" (PDF). Artificial General Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3–6, 2011. Berlin: Springer.
- ^ Soares, Nate (2015). The Value Learning Problem (PDF) (Technical report). Machine Intelligence Research Institute. 2015-4.