> Bot scripting in Dota is done via lua scripting. This is done at the server level, so there's no need to do things like examine screen pixels or simulate mouse clicks; instead scripts can query the game state and issue orders directly to units. Scripts have full access to all the entity locations, cooldowns, mana values, etc that a player on that team would expect to. The API is restricted such that scripts can't cheat -- units in FoW can't be queried, commands can't be issued to units the script doesn't control, etc.
I wouldn't be surprised if they had access to a less restricted AI. Until we see more concrete information I remain sceptical that they are only using the official API.
Why? Everything it needed for that 1v1 was available. There was nothing in the FoW that needed to be known, it can keep timers about enemy cooldowns, gold and xp (approximations at least since it's somewhat random).
The statements are not contradictory - they can have a superior api which doesn't permit 'cheating'. For example they could expose c++ api rather than a lua api for performance concerns. Furthermore some of the api calls (e.g. GetNearbyCreeps) have arbitrary restrictions.
Wow, that was not even close either. And it only takes it two weeks to reach that level of play? OpenAI should create a team of five (as they said they would) and let it play in the qualifiers next year.
I'm not arguing that, just that it would be cool. The creators mentioned on stage that next year it would be a team of five bots. They also said it took the bot two weeks to reach the level it was at right now, so I'm curious to see how/if it'll evolve in the future.
I don't want to be "that guy", but I used to play a lot of LOL, and I feel like 5v5 is in a different ballpark in terms of difficulty than a 1v1. Even champion select is strategically intensive. TSM (one of the best US teams), lost to an amateur team because they got out done in team selection.
I would even bring that to one more level of difficulty. There are so many choices can be made in-game to gain advantages over enemy: item build, gank/farm distribution, resource sharing, vision fight, choose to defense or let go/trade of tower/objectives...
If we can have AI that beat Pro at 5v5 in Dota, it would very likely be an AGI.
To be fair, the 1v1 Shadow Fiend matchup with these rules is A LOT about exact distances and timings. To which the bot has direct API access and the human has to got the whole GPU -> screen -> eyes -> brain -> muscles -> mouse -> CPU cycle.
Agreed, bot will always be good at spell casting, last hitting, which actually is a major factor in winning(the sooner you get higher level you can zone the other player).
Especially for Shadow Fiend, which, for those who don't know, gets extra damage for every last-hit. If you're good at it, you can have extra damage equating to a late game item as an early game character.
Shadow Fiend really comes into its own when blind shots happen, as with many other heroes, fights can be lost to line-of-sight. That requires knowing your opponent's mind and prediction. I can't watch the video just yet, but this is the human behavior you should look out for (such as AlphaGo demonstrated repeatedly in the most recent match).
Given how fast these things generally progress, I wouldn't be surprised if these 5 bots would win the championship in 2018. Or maybe in 2019 at the latest.
This is really cool outreach on OpenAI's part. So many young people watch The International and I'm guessing that at least a few of them are more interested in CS and STEM from seeing this.
Interesting. That means it accumulates the equivalent of 24 * 14 * 300 = 100,800 hours of experience. That's about double the amount of practice one gets for playing 10 hours a day for 14 years.
Not that this isn’t very cool, but 1v1 Dota isn’t anything like the full game, it’s mostly a competition of who has better micro. If it can beat a team of pros at 5v5– which is where the imperfect information, short-vs-long term strategy and inter-agent communication challenges come into play— then I’ll be impressed.
Honestly, you don't even have to make all the AI heroes separate. We automatically assume that each would be controlled by a separate virtual player, but one virtual player could very well control all controllable units on a team. That'd be interesting to watch.
This was fun. It's our intern's last day on the ML infra team here. And he happens to be a competitive collegiate DOTA player. We were all crowded around watching the screen, shouting. Couldn't have planned a better send off. Nice job, OpenAI!
To be clear 1v1 mid lane is harder than 1 unit v 1 unit in sc. But it is at least 3 levels behind 5v5 human play.
So first it's laining, which is showing in the game.
Then is the ganking basically 2-3 player working together.
Above that that's 5 men team fight.
Then you have the strategy planning in the band pick phase, and in game movement coordination.
There are other things like item choices, in game communication etc.
I guess bots never tilt...
This 1v1 match is definitely a proof of the strength and maturity of modern AI.
It will be extremely interesting to see if they can train a bot to achieve the above intelligence. If so, I guess the game is pretty much losing a lot of its appeal.
How does having a bot succeed cause appeal to be lost?
First you're suggesting that the strategy will be "found" and never beaten (no room for improvement, not even in hero selection).
Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily.
> How does having a bot succeed cause appeal to be lost?
You might think not beating bots is OK. Not for me and many people have been involved in competitive scene. The moment machine beats human, a large part of the competition is gone. The essence is that you want to be the best. But if all you can do is to beat some other inferior opponents, what's the point?
> Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily.
Human Chess players lost against Deep Blue in 1997; people still play Chess. In fact, they've learned a lot of strategies from the strategically superior AI.
The micromanagement task in StarCraft Broodwar (with TorchCraft) that has been used by a few people already in the past year to do some RL research has been tackled basically only with #units > 1, because getting a balanced 1v1 is possible only with certain combination of units, and it's indeed pointless otherwise.
Kiting is maybe a notable exception, but it is also restricted to a relatively small group of unit matchups.
I'm not sure about DOTA, but in LOL there is also champion select which happens before the game starts, where you try to pick/ban/steal champions to get a good team comp and get the other team to have a bad team comp. It's lots of know overall strategy, what your enemy likes, and some game theory.
You can lose the game before your summoner even hits the field.
Out of 112 heroes which are currently available in Captain's Mode, 107 have been picked at least once during the group stage and main event. Two (Lion and Tiny) have each been banned once but never picked, and three (Bane, Spectre, and Skele-- er, Wraith King) have not been picked or banned.
Yup, that's exactly the same in DOTA. The pick phase is considered by some commentators to be the most interestingly strategic part of the entire game!
(At a pro level, obviously. At the level I play at, not so much...)
MEH. I am not impressed. Restricting the number of parameters and variables in order to produce a bot that can do 1 sub-set of tasks really well is nothing special imo.
Complexity and simulations of what you have not yet encountered is something humans can do with ease. This is not something a bot can do in a complex environment like DOTA.
I'll be the first to admit that I was wrong and that AI is truly a thing once I see a bot like this one beat a pro-team in a 5v5. Until then, meh...
is an improvement against the other bots, it can be really good for training players, I would love to test it my self, but yeah dota is biggest that this, not only 5 v5, also 100+ heroes, and professional player can play a variety of heroes on mid with different matchups and its nuisances, the bot still need it a lot to even dominate 1v1.
How do things like reaction time, and actions per second work with something like this? Is it just an assumed advantage the ai gets, or does it simulate the limitations of a human? If it doesn't, how big of an advantage is it in an ai versus human competition?
That was pretty much the entire reason it won. It did some standard high-level 1v1 laning techniques, but if human "conditions"[1] were implemented it would look a lot different.
Just the blocking of the lane creeps at the start is already super-human and gives the bot a huge advantage.
[1] like lower actions per minute, latency, occasionally missclicking and not being 100% certain about distances
This will be HUGE for competitive Dota, even if they never take it further.
In much the same ways top chess players have learned from engines, I have to think top Dota players will practice against and study the hell out of this bot if OpenAI makes it available.
Not everything is replicable by humans... for instance I noticed it constantly animation-canceling razes and only finishing it if it was going to hit; a human will definitely mess this up. But other things can definitely be used. It was positioning somewhat strangely, for example.
In the interest of a "fairer" comparison, I wonder how much of a difference it would make to force the AI to simulate mouse/keyboard input and interpret the raw screen buffer output, rather than using direct APIs into the game's guts, to more faithfully emulate its human opponent. I'm guessing the peripheral inputs wouldn't be much of a hurdle, but the image processing step could be very interesting.
Raw screen is not as big of a deal as it would seem, in DotA and Starcraft you can glimpse most of the information you can see on a screen through the minimap.
It would seem to me that this is an oversimplification. Doing screen based video processing is not simple and is not exact especially where camera movements are independent of game dynamics as well as the difficulties inherent in doing fast frame classification- there has been some good work in this direction recently but far from perfect(yolo, ssd). The route deep mind took in partnering with the game manufacturers to gain inputs directly while receiving global information about game state seems the simplest for their goal of training rl agents. We considered doing a project around this but figured it was a massive iceberg.
Oh I misread the "Direct access to game API section", I thought they meant making it so that the machine can only see "what's on their screen", so if someone were to cast a long range skill for instance they wouldn't get that information. Which is a distinct advantage for machines.
That is indeed, by no means trivial. I do think it's pointless though, that's just another exercise for the sake of making it another exercise.
If you're interested I did something similar to what you describe for MarioKart http://kevinhughes.ca/blog/tensor-kart. Its a lot simpler than DOTA though
It's always been prevented in Dota 2. Buildings have backdoor protection, which makes them regenerate lost HP from recent attacks unless there are creeps nearby.
this whole emphasis that openai and deepmind put on human/robot collaboration is a paper thin pr move in my opinion. the robots will be better than any human, humans will not be able to contribute a single thing soon. but they try to make us all feel safe by making it look like they benefit from our brains.
I know it's not the same type of AI, but in chess there's a whole scene for computer + human play. A chess engine on its own can have trouble seeing strategic ideas that humans can recognize (e.g. opposite-colored bishop endgames, certain closed positions, and fortresses) so an engine on its own will lose to that engine being assisted by a skilled human player. In other words, humans are still capable for contributing at least a little. That said, it's not much - I think a chess GM paired with an engine will probably only be able to beat an engine rated ~100 or so points higher than their own.
It will be interesting for deep learning, though, where the ideas are a bit more abstract. Perhaps humans will be useful for a while longer.
AI uses many simulations to train its network. It's appropriate for chess or Dota, but you can't do that for real war, for example, there were not enough wars to learn from them and you can't simulate war good enough. Or making a business plan for Oracle corporation: it's unique situation, you don't have millions of Oracles bankrupting to learn from it. Humanity has this knowledge, but it's encoded in books, teachers and experts. So AI will need to communicate with people to adapt to our society, they can offer advices to experts, but they need to learn from those experts first.
As a longtime DotA player and someone who's following the pro scene, this is very impressive. Especially considering how it's beaten Sumail, widely regarded as one of the best 1v1 players in the world. Can't wait to see what OpenAI have in store a year from now for 5v5.
>The bot didn't recognize items on ground so he expended Mana picked up mango then killed. So yes, he won, but it was more gimping the bot.
That's insane that he figured out how to beat it so quickly. I feel there are other ways to cheese it too. Like maybe survive until 6 -> rush shadow amulet -> smoke -> activate and walk into lane during fade time -> ult when the wave/bot is on top of you. I bet the bot has never seen invisibility and wouldn't know what to do
These types of MOBA games are a good precursor of military unit management. To be honest, I'm not totally sure whether to be happy or sad about this sort of thing. The strategic control of drone units on a combat field, without the loss of personel (on the countries with this technology, anyways) should make me happy, and does, to a point. BUT the potential for abuse (and, no, I haven't even begun to extrapolate towards the Terminator, nightmare scenarios) is rampant. Fewer people with a conscience on the battlefield or controlling the apparatuses of war may or may not be worth the cost of the lives of young men and women..... but I think wars should be fought by old men and women with swords, anyway. If you're old and can look i to the face of your enemy while you kill them, there's a better chance it's worth killing and dying for (by the numbers, anyways).... plus, the President, Congress and the Senate have to serve in combat positions, in my little fantasy scenerio =)
I think it's scarily close to happening, or already has. Just the other day I saw the Drone Racing League on one of ESPN's alternative channels, and all I could think about was how effective they would be as kamikaze bombs. If these people can race drones up to 150+ mph and turn sharp corners through obstacles, then the military is already two steps ahead.
DotA is not a MOBA and it's not a very good precursor for military unit management. I'd say Starcraft is a better comparison, and they are working on applying DeepMind to that right now [0].
DotA and Dota 2 are the games that helped define the term MOBA. I'm not sure why you're saying it isn't one, especially since it's in the first sentence of both Wikipedia articles.
Sometimes people prefer to label dota as an ARTS:
Action Real Time Strategy.
Some people dislike the MOBA nomenclature because it comes from Riot (maker of League of Legends; a clone of dota). In addition the term is pretty vague. Call of Duty, Starcraft, and DotA could all technically be “MOBAS”.
ARTS describes it a little better but I don’t think that term is great either.
In my opinion ARTS doesn't describe it better but introduces confusion since it implies these games have something in common with real time strategy games. I don't think they really have anything to do with RTS despite being originally conceived on the Warcraft 3 engine and assets. At least MOBA makes it clear that it's an entirely separate category.
Managing farming times and item spikes is part of the strategy. Of course it's not a true RTS, but I think that term is a bit better than MOBA; I'm also biased against Riot.
DotA is generally considered to have created the MOBA genre. That being said, you are right that MOBA's aren't as good a comparison for military unit management as games like Starcraft or Warcraft.
Some people dislike that term. Many people in the dota community prefer ARTS over MOBA. Though yes, technically it’s a MOBA; many other games could be MOBAs because it’s fairly vague.
These types of MOBA games are a good precursor of military unit management.
Not really.
There may be some limited use cases where this beats traditional sandboxing, wargaming or milsim. However, the primary distinction is that DOTA and the likes have fixed rule sets (albeit large and complex), and a near infinite number of simulations that can be run, wherein the actual "battle" will conform to those rule sets and play variability.
It may not seem like it, but we (military) only have a tiny slice of possible data from the past few hundred years, that would be appropriate to train a DNN on. Not only that, there are few wars that conform to previous war patterns. There are some constants, and we study those, but they don't really give you what you need for troop movements and tactics the same way as is done in DOTA.
The developer interviewed claimed that there was no domain-specific knowledge and implied (though didn't explicitly state) there wasn't any training against non-OpenAI bots or players. (I'd love to know the reward function they used for whatever Q-learning variation they ran with.)
If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking. (I can't find a good GIF, but this is walking in a wiggly path in front of the first wave of neutrals on your side, delaying their progress and pushing the lane towards you, which is good for ~reasons.)
Creep blocking isn't all that hard in dexterity--I am a terrible dota player and I can more or less do it. And it's one of the most common pieces of dota knowledge; every pro player does it and since it's relatively easy compared to a lot of pro micro, everyone else rapidly learns they should.
But nevertheless--the bot had enough games that it could randomly jump in front of the wave enough times that it noticed a win rate improvement for that slight wave push, and begin to do it intentionally? (And then get good at it?) Damn.
One thing I don't really know about Q-learning and the typical nets used for it: I am guessing it is likely that internally to the bot's evaluation functions, there is some learned feature whose activation correlates well to the location of the wave equilibrium (since that's a feature that correlates well with winning!) At that point, is it likely that the bot can learn in smaller increments--that is, it knows that pushing equilibrium towards itself is good, and thus randomly creep blocking a little becomes reinforced (rather than having to notice the creep block's effect on game wins?)
I expect your conjecture is correct. That's the whole point of deep learning -- there are many layers that automate what would otherwise be human feature extraction.
The magic here doesn't come solely from deep learning, but also from having access to massive simulation. Simulation can make an almost average human-level deep neural net become better than the best human. It happened for Go as well, where AlphaGo learned by self playing millions of games.
I think there is a deep link between simulation and AGI. An AGI would need to be able to imagine how people and objects would act and react in any situation, which is the same as the ability to simulate the world, or to imagine. We might be able to create small simulations like Dota2, but the real world will be much harder.
Yes, I've often thought that too, and really as humans we are running simulations all the time in our head. That's kind of what imagination is.
We run through conversations, physical events / muscle memory, are constantly predicting the world around us - we often don't even remember moving through the environment on regular routes unless something unusual disturbs our running predictions.
It's super interesting to see some of that come through, sure in a more 'brute force' manner, but we randomly brute force bumped through the world as children too, until we pruned our selection trees.
I don't know that they specifically said there was no domain-specific knowledge. iirc they said they didn't "teach it the rules of dota" but they also said the training involved "coaching". I interpret that to include showing the bot useful techniques (like creep blocking) which the AI then learned leads to higher win rates, etc.
They also mentioned that in the beginning the bot figured out that the best way to win was to not play the game (aka hiding in it's own base). Then it started running around wildly, dying to enemy towers in the wrong lanes at the map. So it definitely took some nudging getting it to do something more than being AFK in base.
Barring faction imbalance, there's a 50% chance to win by staying in base. However, I'm not sure how running around and dying could negatively impact your win percentage unless the other bot is also outside of the base.
The idea is that by staying in base you don't die, while your opponent (which is still pretty bad) roams around and dies.
This highlights the importance of gradual adversarial training: if the opponent was perfect, there would simply be no way to win, and hence no signal on how to gradually improve to eventually find the optimal strategy.
Also shows a weakness in most training algorithms today that don't have, for example, the capability to watch their opponent closely and try to mimic it's actions in case he's better than you. Humans do this so quickly you can see Dendi already tried replicating the AI's strategy after 1 loss.
The idea is that first generations run around dying so the optimal response is to just stay put and let the opponent die. This now has basically 100% win rate so both of them do it, lowering it to 50%.
The other part is that there are random variations to the strategy it uses. This it experiments - they won't stay in base forever since that is giving you only 50% winrate (based on whose creeps would push tower first).
This random variation then means that the bot might go out, autoattack a creep wave and then come back to to base. Or anything similar to that.
This new strategy now has almost 100% win rate and the stay in base is no longer viable.
If it went out and died to a tower, it doesn't matter, it can just not use it and there will be period of base AFKing again, but it only takes one improvement (pushing a wave or hitting tower couple times) to make the base AFK obsolete and force new progress.
I was surprised to learn that he is the CTO of Open AI.
> If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking
Same. It is such long set of moves to get a perfect block. To have such a long move-set figured out as something advantageous is very high level of planning from the bot.
I think they either had a reward for creep blocking or they used the "learning from human preferences" paper to coach it for that behaviour. I would be very terrified if it learnt creep blocking on its own with no help.
218 comments
[ 3.3 ms ] story [ 266 ms ] threadBlizzard's Starcraft did the same thing for their recently announced AI project where Starcraft 2 got an API for the AI developers to work with:
https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
> Bot scripting in Dota is done via lua scripting. This is done at the server level, so there's no need to do things like examine screen pixels or simulate mouse clicks; instead scripts can query the game state and issue orders directly to units. Scripts have full access to all the entity locations, cooldowns, mana values, etc that a player on that team would expect to. The API is restricted such that scripts can't cheat -- units in FoW can't be queried, commands can't be issued to units the script doesn't control, etc.
https://www.twitch.tv/dota2ti
I guess I missed it and the stream link is now just a general link to the TI stream, which confused me for a bit.
https://www.youtube.com/watch?v=g7l30-4A6WQ
If we can have AI that beat Pro at 5v5 in Dota, it would very likely be an AGI.
So first it's laining, which is showing in the game.
Then is the ganking basically 2-3 player working together.
Above that that's 5 men team fight.
Then you have the strategy planning in the band pick phase, and in game movement coordination.
There are other things like item choices, in game communication etc.
I guess bots never tilt...
This 1v1 match is definitely a proof of the strength and maturity of modern AI.
It will be extremely interesting to see if they can train a bot to achieve the above intelligence. If so, I guess the game is pretty much losing a lot of its appeal.
You might think not beating bots is OK. Not for me and many people have been involved in competitive scene. The moment machine beats human, a large part of the competition is gone. The essence is that you want to be the best. But if all you can do is to beat some other inferior opponents, what's the point?
> Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily.
This is irrelevant.
Kiting is maybe a notable exception, but it is also restricted to a relatively small group of unit matchups.
See for instance the following papers:
- https://arxiv.org/abs/1609.02993
- https://arxiv.org/abs/1702.08887
- https://arxiv.org/abs/1703.10069
- https://arxiv.org/abs/1705.08926
Full disclosure: I'm one of such researchers, and I have authored some of the papers on the topic.
You can lose the game before your summoner even hits the field.
https://www.reddit.com/r/leagueoflegends/comments/4n5og5/com...
https://www.dotabuff.com/esports/events/197/picks
(At a pro level, obviously. At the level I play at, not so much...)
Complexity and simulations of what you have not yet encountered is something humans can do with ease. This is not something a bot can do in a complex environment like DOTA.
I'll be the first to admit that I was wrong and that AI is truly a thing once I see a bot like this one beat a pro-team in a 5v5. Until then, meh...
Just the blocking of the lane creeps at the start is already super-human and gives the bot a huge advantage.
[1] like lower actions per minute, latency, occasionally missclicking and not being 100% certain about distances
In much the same ways top chess players have learned from engines, I have to think top Dota players will practice against and study the hell out of this bot if OpenAI makes it available.
Not everything is replicable by humans... for instance I noticed it constantly animation-canceling razes and only finishing it if it was going to hit; a human will definitely mess this up. But other things can definitely be used. It was positioning somewhat strangely, for example.
That is indeed, by no means trivial. I do think it's pointless though, that's just another exercise for the sake of making it another exercise.
It will be interesting for deep learning, though, where the ideas are a bit more abstract. Perhaps humans will be useful for a while longer.
https://twitter.com/Phillip_Aram/status/896162260455800832
That's insane that he figured out how to beat it so quickly. I feel there are other ways to cheese it too. Like maybe survive until 6 -> rush shadow amulet -> smoke -> activate and walk into lane during fade time -> ult when the wave/bot is on top of you. I bet the bot has never seen invisibility and wouldn't know what to do
[0] https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
Some people dislike the MOBA nomenclature because it comes from Riot (maker of League of Legends; a clone of dota). In addition the term is pretty vague. Call of Duty, Starcraft, and DotA could all technically be “MOBAS”.
ARTS describes it a little better but I don’t think that term is great either.
Managing farming times and item spikes is part of the strategy. Of course it's not a true RTS, but I think that term is a bit better than MOBA; I'm also biased against Riot.
[0]: https://en.wikipedia.org/wiki/Multiplayer_online_battle_aren...
Not really.
There may be some limited use cases where this beats traditional sandboxing, wargaming or milsim. However, the primary distinction is that DOTA and the likes have fixed rule sets (albeit large and complex), and a near infinite number of simulations that can be run, wherein the actual "battle" will conform to those rule sets and play variability.
It may not seem like it, but we (military) only have a tiny slice of possible data from the past few hundred years, that would be appropriate to train a DNN on. Not only that, there are few wars that conform to previous war patterns. There are some constants, and we study those, but they don't really give you what you need for troop movements and tactics the same way as is done in DOTA.
If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking. (I can't find a good GIF, but this is walking in a wiggly path in front of the first wave of neutrals on your side, delaying their progress and pushing the lane towards you, which is good for ~reasons.)
Creep blocking isn't all that hard in dexterity--I am a terrible dota player and I can more or less do it. And it's one of the most common pieces of dota knowledge; every pro player does it and since it's relatively easy compared to a lot of pro micro, everyone else rapidly learns they should.
But nevertheless--the bot had enough games that it could randomly jump in front of the wave enough times that it noticed a win rate improvement for that slight wave push, and begin to do it intentionally? (And then get good at it?) Damn.
One thing I don't really know about Q-learning and the typical nets used for it: I am guessing it is likely that internally to the bot's evaluation functions, there is some learned feature whose activation correlates well to the location of the wave equilibrium (since that's a feature that correlates well with winning!) At that point, is it likely that the bot can learn in smaller increments--that is, it knows that pushing equilibrium towards itself is good, and thus randomly creep blocking a little becomes reinforced (rather than having to notice the creep block's effect on game wins?)
I think there is a deep link between simulation and AGI. An AGI would need to be able to imagine how people and objects would act and react in any situation, which is the same as the ability to simulate the world, or to imagine. We might be able to create small simulations like Dota2, but the real world will be much harder.
We run through conversations, physical events / muscle memory, are constantly predicting the world around us - we often don't even remember moving through the environment on regular routes unless something unusual disturbs our running predictions.
It's super interesting to see some of that come through, sure in a more 'brute force' manner, but we randomly brute force bumped through the world as children too, until we pruned our selection trees.
But in the future coaching might indeed just be a human expert talking in natural language to teach it how to be better! https://www.youtube.com/watch?v=L_jGDV_5gPA&t=6
You might also pull aggro from their creep wave, which I think should also win you the game in most circumstances.
This highlights the importance of gradual adversarial training: if the opponent was perfect, there would simply be no way to win, and hence no signal on how to gradually improve to eventually find the optimal strategy.
Also shows a weakness in most training algorithms today that don't have, for example, the capability to watch their opponent closely and try to mimic it's actions in case he's better than you. Humans do this so quickly you can see Dendi already tried replicating the AI's strategy after 1 loss.
The idea is that first generations run around dying so the optimal response is to just stay put and let the opponent die. This now has basically 100% win rate so both of them do it, lowering it to 50%.
The other part is that there are random variations to the strategy it uses. This it experiments - they won't stay in base forever since that is giving you only 50% winrate (based on whose creeps would push tower first).
This random variation then means that the bot might go out, autoattack a creep wave and then come back to to base. Or anything similar to that.
This new strategy now has almost 100% win rate and the stay in base is no longer viable.
If it went out and died to a tower, it doesn't matter, it can just not use it and there will be period of base AFKing again, but it only takes one improvement (pushing a wave or hitting tower couple times) to make the base AFK obsolete and force new progress.
"A strange game. The only winning move is not to play."
- AI "Joshua" in "Wargames", 1983
I still can't quite tell if this behaviour was coached to the bot or it figured it out itself.
I was surprised to learn that he is the CTO of Open AI.
> If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking
Same. It is such long set of moves to get a perfect block. To have such a long move-set figured out as something advantageous is very high level of planning from the bot.