OpenAI Five
OpenAI Five is the name of an machine learning project and system that performs as a team of bots playing against human players in the competitive five-on-five video game, Dota 2. The bots learn to play against humans at a high skill level entirely through machine learning. The system was developed by OpenAI, a non-profit American AI research and development company founded with the mission to develop safe AI in a way that benefits humanity. OpenAI Five's development traces back to 2017, where it was first demonstrated in a live one-on-one[disambiguation needed] game against a professional player of the game known as Dendi, who lost to it. The following year, the system had advanced to the point of performing as a full team of five, and began playing against professional teams.
According to OpenAI, the company uses the game as an experiment for general-purpose, applied machine learning to capture the unpredictability and continuous nature of the real world. The team stated that the complex nature of Dota 2 and its strong reliance on having to work together as a team to win was a major reason it was specifically chosen. The algorithms used for the project have also been applied to other systems, such as controlling a robotic hand. The project has also been compared to a number of other similar cases of AI playing against and defeating humans, such as Watson on the gameshow Jeopardy!, Deep Blue in chess, and AlphaGo in the board game Go.
History
Development on the algorithms used for the bots started in late 2016.[1] Before becoming a team of five, the first public demonstration occurred at The International 2017, the annual premiere championship tournament for the game, where Dendi, a professional Ukrainian player of the game, lost against a bot in a live one-on-one matchup.[2][3] After the match, CTO Greg Brockman explained that the bot had learned by playing against itself for two weeks of real time, and that the learning software was a step in the direction of creating software that can handle complex tasks "like being a surgeon".[4][5] OpenAI calls the system "reinforcement learning", as the bots learn over time by playing against itself hundreds a times a day for months, in which they are rewarded for actions such as killing an enemy and destroying towers.[6][7][8] By June 2018, the ability of the bots expanded to play together as a full team of five and were able to defeat teams of amateur and semi-professional players.[9][10][11][12] At The International 2018, OpenAI Five played in two games against professional teams, one against the Brazilian-based paiN Gaming and the other against an all-star team of former Chinese players.[13][14] Although the bots lost both matches, OpenAI still considered it a successful venture, stating that playing against some of the best players in Dota 2 allowed them to analyze and adjust their algorithms for future games.[15]
Architecture
Each OpenAI Five network contains a single layer with a 1024-unit LSTM[16] that observes the current game state extracted from the Dota developer’s API. The neural network conducts actions via numerous possible action heads (no human data involved), and every head has meaning. For instance, the number of ticks to delay an action, what action to select – the X or Y coordinate of this action in a grid around the unit. In addition, action heads are computed independently. The AI system observes the world as a list of 20,000 numbers and takes an action by conducting a list of eight enumeration values. Also, it selects different actions and targets to understand how to encode every action and observes the world.[17]
OpenAI Five has been developed as a general-purpose reinforcement learning training system on the "Rapid" infrastructure. Rapid consists of two layers: it spins up thousands of machines and helps them ‘talk’ to each other and a second layer runs software. OpenAI Five has played 180 years’ worth of games in a self-play technique, which is a part of the reinforcement learning approach, using a scaled-up version of "Proximal Policy Optimization".[18][19]running on 256 GPUs and 128,000 CPU cores.[20]
OpenAI 1v1 bot (2017) | OpenAI Five (2018) | |
---|---|---|
CPUs | 60,000 CPU cores on Azure | 40128,000 preemptible CPU cores on GCP |
GPUs | 256 K80 GPUs on Azure | 256 P100 GPUs on GCP |
Experience collected | ~300 years per day | ~180 years per day (~900 years per day counting each hero separately) |
Size of observation | ~3.3 kB | ~36.8 kB |
Observations per second of gameplay | 10 | 7.5 |
Batch size | 8,388,608 observations | 1,048,576 observations |
Batches per minute | ~20 | ~60 |
Comparison with other AI game systems
Prior to OpenAI Five, other AI versus human experiments and systems have been successful used before, such as Jeopardy! with Watson, chess with Deep Blue, and Go with AlphaGo.[21][22][23] In comparison with other games that have used AI systems to play against human players, Dota 2 differs as explained below:
Long run view: The bots run at 30 frames per second for an average match time of 45 minutes, which results in 80,000 ticks per game. OpenAI Five observes every fourth frame, generating 20,000 moves. By comparison, chess usually ends before 40 moves, while Go ends before 150 moves.[24]
Partially observed state of the game: Players and their allies can only see the map directly around them. The rest of it is covered in a fog of war which hides enemies units and their movements. Thus, playing Dota 2 requires making inferences based on this incomplete data, as well as predicting what their opponent could be doing at the same time. By comparison, Chess and Go are "full-information games", as they do not hide elements from the opposing player.[25][26]
Continuous action space: Each playable character in a Dota 2 game, known as a hero, can take dozens of actions and that target either another unit or a position. The OpenAI Five developers allow the space into 170,000 possible actions per hero. Without counting the perpetual aspects of the game, there are an average of ~1,000 valid actions each tick. By comparison, the average number of actions in chess is 35 and 250 in Go.[27]
Continuous observation space: Dota 2 is played on a large map with ten heroes, five on each team, along with dozens of buildings and non-player character (NPC) units. The OpenAI system observes the state of a game though developers’ bot API, as 20,000 numbers that constitute all information a human is allowed to get access to. A chess board is represented as about 70 lists (an 8x8 board of 6-piece types – here is historical info); where a Go board has about 400 enumerations (a 19x19 board of 2-piece types).[28]
Reception
OpenAI Five have received acknowledgement from the AI, tech, and video game community at large. Microsoft founder Bill Gates called it a "big deal", as their "victory required teamwork and collaboration".[29][30] Chess player Garry Kasparov, who lost against the Deep Blue AI in 1997, stated that despite their losing performance at The International 2018, the bots would eventually "get there, and sooner than expected".[31]
Andreas Theodorou, an AI researcher at the University of Bath who uses computer games to study collaboration, says OpenAI Five was a "big step forward" in the AI industry, although noting that perhaps the most significant achievement was their use of transparent visualizations.[32] In a conversation with MIT Technology Review, AI experts also considered OpenAI Five system as a significant achievement, as they noted that Dota 2 was an "extremely complicated game", so even beating non-professional players was impressive.[33] Inspired by the success of OpenAI five, other AI companies have begun to develop systems that will be able to compete in similar complex video games that require strategic thinking, team play, and reasoning, such as StarCraft.[34][35]
References
- ^ "OpenAI Five". openai.com/five.
- ^ Savov, Vlad. "My favorite game has been invaded by killer AI bots and Elon Musk hype". The Verge. Retrieved 25 June 2018.
- ^ Frank, Blair Hanley. "OpenAI's bot beats top Dota 2 player so badly that he quits". Venture Beat. Archived from the original on 12 August 2017. Retrieved 12 August 2017.
{{cite web}}
: Unknown parameter|deadurl=
ignored (|url-status=
suggested) (help) - ^ "Dota 2". blog.openai.com. Retrieved 12 August 2017.
- ^ "More on Dota 2". blog.openai.com. Retrieved 16 August 2017.
- ^ Simonite, Tom. "Can Bots Outwit Humans in One of the Biggest Esports Games?". Wired. Retrieved 25 June 2018.
- ^ Kahn, Jeremy. "A Bot Backed by Elon Musk Has Made an AI Breakthrough in Video Game World". Bloomberg. Retrieved 27 June 2018.
- ^ Clifford, Catherine. "Bill Gates says gamer bots from Elon Musk-backed nonprofit are 'huge milestone' in A.I." CNBC. Retrieved 29 June 2018.
- ^ "OpenAI Five Benchmark". blog.openai.com. Retrieved 25 August 2018.
- ^ Simonite, Tom. "Can Bots Outwit Humans in One of the Biggest Esports Games?". Wired. Retrieved 25 June 2018.
- ^ Vincent, James. "AI bots trained for 180 years a day to beat humans at Dota 2". The Verge. Retrieved 25 June 2018.
- ^ Savov, Vlad. "The OpenAI Dota 2 bots just defeated a team of former pros". The Verge. Retrieved 7 August 2018.
- ^ Simonite, Tom. "Pro Gamers Fend off Elon Musk-Backed AI Bots—for Now". Wired. Retrieved 25 August 2018.
- ^ Quach, Katyanna. "Game over, machines: Humans defeat OpenAI bots once again at video games Olympics". The Register. Retrieved 25 August 2018.
- ^ "The International 2018: Results". blog.openai.com. Retrieved 25 August 2018.
- ^ "Understanding LSTM Networks". colah's blog. Retrieved 27 August 2015.
- ^ "OpenAI Five". blog.openai.com. 25 June 2018. Retrieved 25 June 2018.
- ^ "Why are AI researchers so obsessed with games?". QUARTZ. Retrieved 4 August 2018.
- ^ Schulman, John; Wolski, Filip; Dhariwal, Prafulla; Radford, Alec; Klimov, Oleg (2017). "Proximal Policy Optimization Algorithms". arXiv:1707.06347 [cs.LG].
- ^ "OpenAI Five". blog.openai.com. 25 June 2018. Retrieved 25 June 2018.
- ^ Gabbatt, Adam (17 February 2011). "IBM computer Watson wins Jeopardy clash". The Guardian. Retrieved 17 February 2011.
- ^ "Chess grandmaster Garry Kasparov on what happens when machines 'reach the level that is impossible for humans to compete'". Business Insider. Retrieved 29 December 2017.
- ^ "DeepMind's Go-playing AI doesn't need human help to beat us anymore". Verge. 18 October 2017. Retrieved 18 October 2017.
- ^ "OpenAI Five". blog.openai.com. 25 June 2018. Retrieved 25 June 2018.
- ^ "OpenAI Five". blog.openai.com. 25 June 2018. Retrieved 25 June 2018.
- ^ "A team of AI algorithms just crushed humans in a complex computer game". MIT Tech Review. Retrieved 25 June 2018.
- ^ "OpenAI Five". blog.openai.com. 25 June 2018. Retrieved 25 June 2018.
- ^ "OpenAI Five". blog.openai.com. 25 June 2018. Retrieved 25 June 2018.
- ^ "Bill Gates says gamer bots from Elon Musk-backed nonprofit are 'huge milestone' in A.I." CNBC. 28 June 2018. Retrieved 28 June 2018.
- ^ "Bill Gates hails 'huge milestone' for AI as bots work in a team to destroy humans at video game 'Dota 2'". Business Insider. Retrieved 27 June 2018.
- ^ "Gary Kasparov's Twitter". 24 August 2018. Retrieved 24 August 2018.
- ^ "AI bots trained for 180 years a day to beat humans at Dota 2". Verge. 25 June 2018. Retrieved 25 June 2018.
- ^ "A team of AI algorithms just crushed humans in a complex computer game". MIT Tech Review. Retrieved 25 June 2018.
- ^ "Humans Are Still Better Than AI at StarCraft for Now". MIT Tech Review. Retrieved 1 November 2017.
- ^ "CAN BOTS OUTWIT HUMANS IN ONE OF THE BIGGEST ESPORTS GAMES?". WIRED. Retrieved 25 June 2018.