Sometimes it feels like there's a secret competition to see who can author the next infamous "No wireless. Less space than a nomad. Lame."-style comment.
I saw that comment when it was first posted on Slashdot all those years ago. The original iPod was overpriced and it only worked on Macs with FireWire. It really didn't take off until they had a USB model working on Windows at a more reasonable price.
Instead of coding the bots with the rules of Dota 2, they’re thrown into the game and left to figure things out for themselves
I saw a strange behaviour. In the second game, I saw witch doctor use "Maledict" on neutral creeps, but this skill only affects enemy heroes. This only waste mana, put the skill in cool down and have no benefit. How can AI learn it?
Because neural networks are noisy, and they cause small mistakes and misunderstandings like in humans. The AIs concept of an enemy obviously doesn't have as strong of a distinction between "hero" and "creep" (and for that matter "enemy" and "neutral") as those of us who have read the game rules might have because of your quoted statement.
As a further analogy I often see humans make ridiculous and insane leaps of logic (how assignment works when you first learn programming; free speech applies to things besides governments; etc.), and we run on the same style of hardware.
To an extent this demonstrates how far we are from generalized AI. All the Open AI system is is a very specialized animal, that's lived through millions of simulated generations very quickly. It doesn't understand language or concepts - we can't even begin to tell it the rules of the game, only though simulated evolution can we teach it - or even the self awareness many animals do. It's likely closer to the nervous system of an insect than it is to the brain of a dog.
> First, OpenAI could have created a vision system to read the pixels and retrieve the same information that the bot API provides. (The main reason it didn’t is that it would have been incredibly resource-intensive.)
Mmmm, no. Reading the pixels would not retrieve the same information that the bot API provides, because the screen does not contain all that information at any given time.
Any Dota 2 player will know that they don't have access to all this data simultaneously:
It would have to remember the data that is off-screen. It's quite possible to train a recurrent network to do this. Some Atari games require this to play well, like Pacman requires remembering where the ghosts are when they're blinking.
More complicated is the logic to know what to look at when, by keeping track of which parts of the state might be important and out of date, and controlling the viewpoint to update it. I haven't seen a good example of this.
True... well, almost, since data from previous frames becomes stale with time.
I agree that learning how to retrieve the info would be another rather tough problem (on top of all the tough problems the current incarnation of OpenAI's bots already need to solve).
Yes, but then after reading data, they could train a system to remember the information, and decide which information to look at next, and predict their values over time from what inputs it gets, like humans do.
It's not just image recognition though. Gathering relevant information requires making non-trivial decisions about what units to click and how to move the camera around in an useful way. Those are definitely interesting problems.
Isn't it the case that many behaviors had to be specifically encouraged or coded into the bots to get them to work at all? They learned an impressive amount on their own after being pushed in the right direction, but this is far from a pure random reinforcement learning approach as this article suggests.
This article also really understates how clowny the bots were playing. They appeared confused and aimless for much of the game, until they found a clear objective like an enemy hero that moved too close to them, or they had a numbers advantage that allowed them to pressure an enemy building. They were brutal at closing gaps and pursuing enemies to kill without hesitation, but didn't perform well at all beyond that.
This isn't to take away from the accomplishments of OpenAI - to get to this stage after 18 months of work is an impressive engineering feat. It's just telling that the bots have only learned one single strategy, don't even execute that very well, and still have to play with hero restrictions that rule out counters to that specific strategy of aggressive skirmishing and pushing.
Looking "clowny" isn't something that RL agents learn to avoid. You see it even in Go agents. The MCTS algorithm looks at probability of winning, and if it's ~100% regardless of the next move, it'll often waste moves rather than trying to win as quickly as possible.
My understanding is that there are behaviors that are coded into the bots at first to bootstrap their learning capabilities, which are then removed after a certain amount of learning has taken place. In another article that was posted here a few days ago one of the OpenAI team talks about removing the last of these "scripted" capabilities as part of the improvements they plan on making to the bots after this round of games against human players.
I kind of liken it to a human who learns math - you start off by learning the rules (addition, subtraction, algebra, etc.) and work within those confines before you start to use these basic rules to push the boundaries and explore (proofs).
Some behaviors did have to be encouraged, but from what I understand, the behaviors were encouraged through small indirect nudges instead of specifically adding code to the bots to tell them a certain behavior is good. For example, the devs said that the bots never tried to fight Roshan on their own because he does a lot of damage, has a lot of health, and the bots don't know what the reward for killing him is. To encourage the bots to try it, Roshan was given artificially low health so the bots would kill it to see what happens. Once the bots learned that there are rewards for killing him, the devs started to randomize his health between very high and very low values so the bots could get a feel for when they are strong enough to try and kill him during a normal match.
> It's just telling that the bots have only learned one single strategy, don't even execute that very well, and still have to play with hero restrictions that rule out counters to that specific strategy of aggressive skirmishing and pushing.
I suspect this has to with the fact that the bots were designed to be agents with no leadership. As you say, the bots performed very well in skirmishes, they lost because they didn't really know what to do beyond that.
They might fare better if one of the bots could act as a team captain and provide high level instructions to the team, i.e., assign people to harvest, defend, or push.
It seems like that's what the human teams have over the bots -- somebody telling the individual team members what to do.
The bots might be running 5 instances of the software without communication, but they all have the same vision. It's probably better to think of it as one instance controlling 5 bots, since each instance can predict the others perfectly.
I think they should be training 5 different bots for the 1-5 positions because their priorities and roles are quite different. It seemed like all 5 bots had the same, "kill enemy, hit building" strategy, which is locally optimal, but globally harmful (sort of a tragedy of the commons scenario).
Generally, captains in Dota teams don't give specific instructions on what to do when not making specific plays. Carry players will default to returning to farming as efficiently as possible given the current state of the map. Support players will default to placing wards for map vision, stacking camps for the carry to farm, and searching for potential kills. But there definitely is a need for a shot-caller, and OpenAI has no idea how to play that role.
OpenAI 5 had a number of glaring flaws in what it learned. It seemed to have never come to the conclusion that certain characters would have better outcomes if they focused on certain activities. Every character farmed, any character would grab the Aegis (reward item) after killing Roshan (usually this is reserved for a character likely to be in the middle of the fray, dealing a lot of damage).
Another problem was that, until the last few weeks before its matches against pro players, they had allowed both teams to have 5 couriers. This allowed OpenAI 5 to keep up relentless pressure by constantly bringing themselves consumable health regeneration items. This was so unlike a regular game of Dota that the community (including their opponents) complained because normally the courier's time is a valuable resource, and the courier can normally be killed if it is used too close to enemies. With an endless supply of healing items with no risk involved, it didn't even resemble a game of Dota. They did away with this for TI, and it revealed a serious weakness in their relentless-aggression strategy.
I get the impression most of OpenAI's games vs itself ended very quickly, with one team making a mistake during the relentless-aggression early game pushes. As a result, it seemed to have no idea what to do differently as the game went on.
This makes sense. They should add a NN feature called farm priority ranked from 1 to 5 (similar to the one humans use) with 1 being carry, 5 being support, and each taking progressively less farm and lower priority taking Aegis. They could start out with this feature randomized at first, and see whether the same heroes humans prioritize farm on get prioritized by the AI.
I'll own up to wanting to chalk this up to "AI wisdom." The feed-the-carry strategy has always seemed like precisely the same kind of premature optimization that makes AI often easy to beat.
Well the point is that practice actually is enough. Humans have a tendency to think they are special because they can do X. Then a robot does X, and suddenly the goal posts move. Humans are just specialized neural networks plus meat.
Frankly, superior reaction time and other mechanical skills in Dota2 (or Starcraft, or other games which are being tested) makes me think that a typical AI would beat a human.
That's the "gloves come off" AI. Granted, its a problem solved by raw, mechanical button-pushes at a rate beyond what is possible on a keyboard/mouse, but it demonstrates the greatest skills an AI or computer has.
Of course, no one wants to see an AI beat a human using known methodologies. They want to see an AI win at the abstract game of "a game of strategy".
From this perspective, I think the poker AI results over the past year or two have been far more interesting. AIs figuring out the Nash Equalibrium and playing bluffing games around it.
Sure, its a bit mechanical and mathematically inclined, but generalizing and solving the bluffing game is IMO far more useful.
And that list is incomplete if you want to include human high scores on non-emulators (not included in that list). This is even after the reinforcement learning algorithms have been given orders of magnitude more training time than humans. Furthermore, much of the machine performance over humans can be attributed to better reaction time.
IIRC, you can easily build a super-human Atari bot. Just add some off line planning (e.g. paper by Guo), or add manual rewards or features.
No one is doing since there is no point. We want to study Atari WITHOUT these "cheats", so that we can then apply these algos in more complex situations.
I think the OP was using "machines" as reinforcement learning (or whatever the best general learning algorithm). It doesn't count if you program a solution for each individual game. These are learning algorithm benchmarks not programmer benchmarks.
For generalized reinforcement learning algorithms. Humans almost always beat the "machine". The only cases where machine wins is on 100% hand eye coordination and duration.
All ML algorithms are terrible at figuring out how to plan de-novo. So anything that requires multiple contexts or planning is a fail.
>> The more important question might be: can we ever have a fair fight between
humans and machines?
Fair or not fair, there is no API to the real world. If we are going to create
machines that can learn and think outside of simulations, however complex they
may be, these machines will need a way to interact with the physical
environment.
Gary Marcus is on the right track here. Building systems that can "handle the
complexity of the real world", as per OpenAI's stated goal according to the
article, is incredibly ambitious and pointing out the vast distance separating
the current state-of-the-art from that lofty goal is, well, only fair.
I mean, if you think about it, back in the '70s, in the original AI Winter,
one of the big criticisms of AI research was that it languished in simulated
environments like blocks world and didn't perform nearly as well in the real
world. And here we are today, celebrating a bright step on the path to
conquering yet another simulation.
Our simulations have gotten way better in 40 years. You could teach an AI the basics of simulation driving or flying that would work as a baseline for the real world.
Unfortunately, modern, statistical machine-learning based AI is extremely bad at generalising from one environment or one context to another. And, training one some task in a simulation and then performing the same task in the real world requires a very strong ability for generalisation. That is because it is extremely expensive to simulate the real world with any fidelity and therefore every simulation is "cutting corners" - and really, really big corners at that. This gap, between reality and simulation must be covered by generalisation, but our machine learning systems genearlise too poorly.
As a result, training in simulated environments doesn't help handle the full complexity of the real world. Even if we had robot bodies that could move as freely and manipulate objects with as much dexterity as they can do in simulations.
It's even hard to tell whether that robot hand pushing the puck is actually "generalising" like the article claims. Maybe it is, maybe the task is set up so that it makes it possible for the robot hand to push around the puck even when it's on a bag of crisps. It's very difficult to know for sure.
> At least one previously undiscovered game mechanic, which allows players to recharge a certain weapon quickly by staying out of range of the enemy, has been discovered by the bots and passed on to humans
I assume they're talking about blink dagger, and something more advanced than "don't take damage"?
it was discovered that if you stay out of vision and cast raze, the other player does not get stick charges. that was the 1v1 shadow fiend bot a year ago though.
If that's the case, that's incredibly misleading. Especially because that's not undiscovered by human players, just by the OpenAI team, if that's what it is: https://blog.openai.com/more-on-dota-2/
"Sumail pointed out that the bot had learned to cast razes out of the enemy’s vision. This was due to a mechanic we hadn’t known about: abilities cast outside of the enemy’s vision prevent the enemy from gaining a wand charge."
That mechanic was in the patch notes for dota 6.69 released in 2010. Maybe its importance to high level 1v1 shadow fiend was revealed but that's stretching it. It is cool that the bot rediscovered it though.
Doubt that's what they are talking about if the mechanic was undiscovered before this year. Stick + vision mechanics have been well known for a while, I learned about it when I read about the Kunkka mid matchup. Might have been Pajkatt buying mangoes and using them out of vision last year? Not sure, my memory of the details is fuzzy.
From the description it does sound like blink dagger and the range here refers to radiance or necro's heartstopper. It's definitely not "previously undiscovered". Also, the article makes it sound like we saw another AlphaGo's 3-3 invasion, 5th line shoulder-hit kind of moment. We did not. This is more similar to AlphaGo and Fan Hui match, except imagine AlphaGo lost to Fan Hui. The bots did make a lot of interesting moves in 5 invincible chicken meta. The bots appeared very weak in normal meta (constantly check rosh for no reason, inefficient use of ults, don't get me started on warding, etc)
I noticed that bots developed this strategy of using salve while being hit by the tower. Humans used this while tower diving in several games to kill a enemy with low HP. Casters mentioned that - it was first used by OpenAI but I have no way of confirming that.
>> Such a thing does not exist because machines think like humans in the same way
that planes fly like birds.
Could we please lay this empty platitude to reast? "Planes don't fly by
flapping their wings like birds, so why should computers think like humans?".
Well, except that we were only able to built flying machines (rather than
floating ones) when we figured out why birds can fly [1].
And then, the whole of our computer science is based on the idea that a
computer is, actually, the same kind of system as a human mind- a
computational device, a machine that can compute everything and anything that
can be computed. This is the deep insight that informs the ambition to create
artificial intelligence: that brains are a kind of computer, computers are a
kind of brain, and they can both compute the same kind of program, in other
words- intelligence.
Though we may not know how human minds work exactly and therefore we can't
readily copy them, it is thanks to Turing's and Church's insight that we even
have computers today. And so, in a very real sense, we can only ever create
thinking machines that do intelligence in the same way that humans do
intelligence [2].
_______________________
[1] I mean, a boomerang is really an airfoil so I guess Australian Aborigines
had figured out flight long before the Wright brothers, but I gues they didn't
need flying vehicles back then?
[2] Unless there is a paradigm shift and we figure out a better way to do it
etc etc disclaimer disclaimer.
Neural networks are based on human brains. So, how are they both different? It is a very hard question, because if we would understand better the brain we will simulate it better in the computers, how do we know we are not already doing a good simulation of the brain? Any answer to this question would be a step forward to improve our artificial intelligence systems.
The point is that human minds and computers are both computational devices of equivalent power and that we believe that creating artificial intelligence is possible because intelligence is a program that can be computed by a Turing machine just as it is computed by a human mind.
Obviously, the Turing-Church thesis didn't have anything to do with brains. It does, however, have very important implications about our ambition to create human-like artificial minds.
There may be an issue with running a "brain" program on a non-brain hardware depending on what the program is. We don't know what the program is so it's hard to speculate but, it is possible that brain program is not well fitted to silicon based architectures. It may turn out that low-level physical and chemical processes that are crucial to brain functioning are prohibitively expensive to simulate. Brains after all operate in the perfect simulator and evolution had no reason to avoid expensive to simulate solutions. I would guess that it is rather unlikely, but it is possible since even a single cell chemistry, well even a single organelle chemistry is complex. Simulating processes involving RNA may turn out to prohibitively expensive.
Also, I'm sure that theoretically optimal "intelligence program" on silicon processors is vastly different from organic ones. They may share a lot, but the constraints are just so vastly different including the most important ones like energy efficiency or space availability.
>> There may be an issue with running a "brain" program on a non-brain hardware depending on what the program is.
That's not impossible! For example, it's easy to find algorithms that a computer can carry out without error that a human mind would really struggle with. Although this is a case of computational resources, rather than the expressive power of the computational apparatus, it's still the case that it's not always possible to find programs that both humans and computers can compute in practice.
So it may even be that, while computers can run some programs very efficiently, that humans can't, it's the other way around also and computers can't efficiently run the programs that humans can.
In which case of course, either the entire AI enterprise is doomed to failure, or we get lucky and there is some other way to do intelligence that is available to computers but not humans. Who knows!
Human minds are not "computational devices". They are biological organs.
Our current computers function almost nothing like human brains.
All we can say, is that we can write some programs now that give similar output to humans for certain well defined, limited problems. But even then, the mechanism via which those outputs are produced are very, very different.
The mistake is assuming machines that can perform tasks better than humans, will be essentially "like" humans in some fundamental, profound way. There is no reason to believe this is true. These machines may be able to mimic us very well, but their true underlying nature, their goals, ambitions, subjective experience, and values could be very, very different from us in ways that could be very unpleasant for us.
>> Our current computers function almost nothing like human brains.
With respect, but I never said anything about computers functioning like human
brains. What I said is that computers and minds can compute the same class of
programs. I make no assumptions about underlying function.
>> The mistake is assuming machines that can perform tasks better than humans,
will be essentially "like" humans in some fundamental, profound way.
I never stated any similar assumption either.
I think you're responding to some opinion that I didn't express.
> Well, except that we were only able to built flying machines (rather than floating ones) when we figured out why birds can fly [1].
> [1] I mean, a boomerang is really an airfoil so I guess Australian Aborigines had figured out flight long before the Wright brothers, but I guess they didn't need flying vehicles back then?
What? I'm not sure anyone would count a boomerang as "figuring out" flight let alone how birds fly.
I'm no historian but I'm pretty sure flight was first figured out mechanically through trial and error well before we had a good theoretical model for flight, let alone figuring out how birds fly. Either that or we have drastically different meaning for what it means to figure something out.
It took ages for us to fully understand how flight worked, well after we could fly. That's the basis of the apocryphal story:
>> Already, the training infrastructure used to teach the OpenAI Five — a system called Rapid — is being turned to other projects. OpenAI has used it to teach robot hands to manipulate objects with new levels of human-like dexterity, for example. As always with AI, there are limitations, and Rapid isn’t some do-everything algorithm.
Oh, yes, indeed, there are limitations. The robot hand in question can only manipulate cubes and then only a specific kind of cube with standard dimensions, as far as it's possible to tell from all the demonstrations publicised by OpenAI. And, I'm guessing, if they had a robot hand able to play the yo-yo, they wouldn't hesitate to show it.
These are the latest wargames instalments (you know, the Matthew Broderick kool-aid from his 1983 film and so on) and still in their infancy for optimising local, transient tactics behind the overall goal. AIs playing Civilization by Sid Meier vs themselves is fascinating enough and possibly useful for transfer learning here? At the very least, the weakest links in real mid-game situations coming from learned-led plans would be discovered. Apart of military uses of such studies and many other techniques giant corps masquerade behind emoticons and facial recognition apps, why not let the latest AIs play vs climate change, for example? It would be much more fun in general and possibly able to highlight the contribution of many impending agents humans can’t fathom already.
As someone who played a bunch of Dota2 (and was present at TI where this happened), the bots were a joke -- and the humans that played against them were not only super sandbagging and screwing around, they quickly figured out the bots' preferred strategy (deathball) and came up with an effective counterstrategy (split push) that the bots responded very poorly to.
The bots excelled at things you expected computers to be good at, consistently quick reactions/mechanical skill and coordinated skirmishes. The bots fell flat on strategy, optimal resource usage (short and long term), and most importantly, coordinated decision-making.
So as an result, they dominated the early game which has always been heavily mechanics favored and absolutely fell apart mid/late-game when decision-making mattered (and often made decisions that benefited early game economy at the cost of mid/late game).
Computers being good at mechanics is not an impressive result.
Calling them a 'joke' is pretty extreme. Did you watch the benchmark? They were pretty much steamrolling the caster team.
With all due respect, the professionals who have played against the bot have been pretty impressed with previous iteration. The major roadblocks at TI were probably that the bot did not efficiently learn courier mechanics and that the draft was not done by the bot.
There is no mechanical advantage, they have their response time capped at 200ms which is around an elite player's reaction. They can definitely strategize for ganks and the future reward function anticipates high reward roaming/ganking plays.
They have 200ms reaction time without needing to move a mouse and/or target the opponent.
A human has ~200ms simple reaction time, then begins to move their fingers. Which then moves a mouse to target an area, which then requires hotkeys to be pushed. etc. etc.
Its still definitely a mechanical advantage, and it seems evident in the gameplay videos. The bots still seem "super-human" in their aim and some of the timing of spells.
Not to mention they have exact numbers to help, too. A human sees a healthbar ~1/4 full of a hero with ~1200 health and has to guess whether their ~300 damage spell will kill them. If not, are they sure they're in range for an auto attack, too? Will the enemy health regen push them over the life-death boundary in the time it takes for the projectile to reach? The bots know these numbers _precisely_.
Of course there is, no need for button presses, no need for mouse movements. While you can probably train yourself to have a reaction time of 100ms (in case of expected events), executing takes more time. You said you know Dota, but that statement makes it hard to believe.
200ms is faster than the 300ms cast time of Axe's call and Tidehunter's ravage. Pros almost never react fast enough to blink/euls dodge those skills (0s cast time for blink/euls, 200ms reaction time gives the bots a 100ms window to respond). The bots dodged those skills every time. That was the only initiation option available in that draft until late game and the bots were more or less immune to it. That's a pretty significant mechanical advantage.
As you said, the common Axe blink-call combo does not work on these bots; they dodge it immediately.
What was interesting to me was that since the bots trained against themselves, they discarded this combo as being ineffective - the result being that they did not use this combo even against vulnerable human players.
This means that Dota 2 was balanced around the expectation that human's best reaction would be at least 400ms, because I've never seen Axe's blink calls being dodged unless the player had vision of Axe before the blink and thus was expecting it.
The caster team played against the iteration of the bot that was trained on a different version of the game that's very unlike the normal game (multi-invincible couriers).
Multiple invincible couriers makes the game very similar to the 1v1 show match last year where the strategy of just out-live the opponent wins. The casters have never played that version of the game before, and wouldn't know to use that strategy.
The more interesting part was, the bots attempted to use that very same strategy in the single-vulnerable normal version of the game which was drastically less effective; you could only ferry out consumables at 1/5th the rate as before and the courier could be killed so it wasn't a riskless strategy like it was before.
I mean yeah, this version of the bot had drastically less time to train. Only 6 days compared to a few weeks for that version. I know how the game works, I mentioned the couriers already.
I 100% disagree that the invincible couriers make the game "very similar" to the 1v1. There's many more layers of engineering that went into that version. The bots had to learn drafting, cooperation, highground pushing, Roshan, vision, invisibility, etc.
I'm amazed you can trivialize all this as a 'joke'.
If you look at network structure, it acts as one agent, not five. So, free coordination. (See: https://t.co/GPKHPsIu1C)
In my opinion, what i see is a very good player who knows how to chain stun precisely without any strategic depth. If you claim to have built an AI system, which you ultimately want it to evolve to AGI, you at least expect some sort of strategic decision making at the macro level. Though since it has almost perfect micro, it can easily outweight the most of teams. So yeah, with that expectation I see this as a joke, too.
P.S. The model is trained with 128k cpus and 256 gpu. It is able to play 180 years worth of game in a day. Think about it.
It's five independent agents. The article on OpenAI's website and the network structure both say this. I'll zoom in since it's a complicated structure.
They use a hyperparameter called team spirit to cooperate. I don't think the goal of this is AGI at all, so I don't see why people are making that leap. But sure, for the geniuses of HN this must clearly be trivial.
It's not independent agents. The neural networks have the same input, share weights, and also share some activations. With that much sharing, it's better to think of it as one neural network which has output heads for all the 5 players. So coordination is free. Actually, coordination makes as little sense as saying that multiple neurons in a neural network are cooperating, or that the two legs of a humanoid are cooperating to walk. Further there is no game being played between heroes of the same team. They literally have the same objective. The "coordination" buzzword is just another attempt by OpenAI to confuse and mislead readers, and give a false sense of their progress.
They cannot share the same inputs unless the team spirit hyperparameter is exactly 1, which it is not. You are partially correct in that the agents consume the parameters of the four other agents, but it is weighted differently accorsing to team spirit parameters.
The team spirit hyperparameter is a crutch they've introduced themselves. Ideally it should be one. In Dota there is only one objective for the entire team and that's to win the game. The fact that they shape rewards is an implementation detail and doesn't change the fact that Dota 2 does not require cooperation, because there's no cooperation game being played. It's a purely zero-sum adversarial game being played between two teams.
"very similar" in terms of getting regeneration all the time. 5 couriers means deathpush. Simple mistakes from the humans scale up, games won't last long (around 20 minutes), so no long-term strategy is needed. It's not fun to watch some mechanical skill outplaying humans. You can compare it to aimbots in ego-shooters. Yes they aim better, but that's it.
I think they should have trained with 1 courier right from the beginning.
5 couriers also meant hyper-aggressive behavior in the laning stage, because regen is so easy to come by.
If I had to guess, they started out with the 5-courier setup because otherwise the bots would end up fighting over the courier -- just like a typical pub team. :)
There are many things you could do to make the bots not fight for courier. Just implement a queuing system with a priority for cores. Or better yet, make the couriers not usable if another hero is using it.
The way I would guess this played out was that OpenAI tried multiple variants of the game, and this version ended up being good enough to beat human players. Of course, they later learnt that the bots were winning not because they were good, but the variant was imbalanced and its strategies unfamiliar to humans.
Salves are not infinite regen and regen-heavy deathball strats with high sustain have been a thing in the past. Calling it unfamiliar makes it seem like it's a completely foreign, unexplored strategy.
That's not the point. The point is that the humans wouldn't know that regen-heavy deathball strat is optimum unless they've played that game a few times. Since multi-invincible-courier game was not available to the casters, they couldn't really prepare for it or even had any incentive to. They were paid to just play a game, not to win it, and thus they played standard Dota 2, and didn't bother theory-crafting strategies for how to optimize this weird subset of Dota.
> The way I would guess this played out was that OpenAI tried multiple variants of the game, and this version ended up being good enough to beat human players
No. We had implemented scripted courier logic for 1v1, and when switching to 5v5 the easiest starting point was to run five copies of that logic. (Dota's Turbo mode also has five invulnerable couriers.) In June, we had more important restrictions to remove — such as wards & Roshan; you personally were focused on the particular heroes we'd chosen: https://news.ycombinator.com/item?id=17392455. The couriers only become our most important restriction in August.
Maybe from an AI perspective, it's impressive that the bots almost managed to learn the rules of the (still drastically simplified subset) game. From a game skill point of view, it's a joke; it's incredibly obvious that the bots were only able to prolong the game due to consistently better than human mechanical skill.
Of the list of skills you mentioned that the bots "learned", on the main stage, the draft was provided by humans, and from a significantly limited set, the bots never pressured highground, checked for Roshan when it was impossible for it to be up, had nonsensical placement of vision (placed vision where vision already existed), continuously invested in anti-invisibility consumables even when the opposition team had no invisibility.
If the mechanical skills were toned down to human levels, I don't think the latest iteration of the bot on the latest ruleset can even compete with the average human player, much less the pros.
As far as we've seen the bot only knows a single strategy. In the mid game, group, push lanes, push lanes, push lanes. The games vs the caster team had multiple differences from standard dota that allowed this strategy to be effective, such as the multiple invincible couriers (no need to stop your push if you have infinite heals) and a lack of split push ("rat") heroes allowed that would force the ai to abandon their pushes.
The games they won weren't actual dota games - they were a simplified subset of dota that overweights teamfights and mechanical skill - exactly the subset that we'd expect an AI to be best at.
Also the didn't the human team have full control over what characters were used? Doesn't say much that the bots weren't able to strategize late game when they weren't allowed to pick their optimal lineup. The bots should have got to pick not the other way around.
They reversed the characters used for the 2nd game. The pros considered the matchup to be fair (no draft advantage to either side).
It's argued that drafting is one of the major components of competitive play and more often than not, draft advantage wins games, but that wasn't a factor in this match.
This is why we should be way more interested by future research on StarCraft 2 by OpenAI.
This game favors Macro over Micro oven more than Dota2 and the original StarCraft.
Micro has a general advantage of growing at a rate of N^2. That is, 10 Zealots, with perfect micro, can perform roughly as well as 100 Zealots with the worst micro possible.
Ex: 100 Zealots come in one-at-a-time vs the 10-zealots in an inverted-V shape. The 10-zealots will defeat roughly 100.
The bigger and more complicated the board gets, the more and more micro becomes favored.
Was the full list of heroes supported in theses showmatches? Or was this also restricted to a subset of, say, 15 out of 150 (or whatever that number is)?
The announcers/casters were a bit ambiguous about this, AFAIK they said something like : "the teams agreed on a predefined, balanced list of heroes before the game...", and there was no drafting.
(I'm not native speaker, and I don't know much about Dota2 rules)
93 comments
[ 0.26 ms ] story [ 156 ms ] threadDon't think match win would really mean something. Only from the marketing perspective.
I saw a strange behaviour. In the second game, I saw witch doctor use "Maledict" on neutral creeps, but this skill only affects enemy heroes. This only waste mana, put the skill in cool down and have no benefit. How can AI learn it?
As a further analogy I often see humans make ridiculous and insane leaps of logic (how assignment works when you first learn programming; free speech applies to things besides governments; etc.), and we run on the same style of hardware.
To an extent this demonstrates how far we are from generalized AI. All the Open AI system is is a very specialized animal, that's lived through millions of simulated generations very quickly. It doesn't understand language or concepts - we can't even begin to tell it the rules of the game, only though simulated evolution can we teach it - or even the self awareness many animals do. It's likely closer to the nervous system of an insect than it is to the brain of a dog.
> First, OpenAI could have created a vision system to read the pixels and retrieve the same information that the bot API provides. (The main reason it didn’t is that it would have been incredibly resource-intensive.)
Mmmm, no. Reading the pixels would not retrieve the same information that the bot API provides, because the screen does not contain all that information at any given time.
Any Dota 2 player will know that they don't have access to all this data simultaneously:
https://s3-us-west-2.amazonaws.com/openai-assets/dota_benchm...
More complicated is the logic to know what to look at when, by keeping track of which parts of the state might be important and out of date, and controlling the viewpoint to update it. I haven't seen a good example of this.
I agree that learning how to retrieve the info would be another rather tough problem (on top of all the tough problems the current incarnation of OpenAI's bots already need to solve).
This article also really understates how clowny the bots were playing. They appeared confused and aimless for much of the game, until they found a clear objective like an enemy hero that moved too close to them, or they had a numbers advantage that allowed them to pressure an enemy building. They were brutal at closing gaps and pursuing enemies to kill without hesitation, but didn't perform well at all beyond that.
This isn't to take away from the accomplishments of OpenAI - to get to this stage after 18 months of work is an impressive engineering feat. It's just telling that the bots have only learned one single strategy, don't even execute that very well, and still have to play with hero restrictions that rule out counters to that specific strategy of aggressive skirmishing and pushing.
I kind of liken it to a human who learns math - you start off by learning the rules (addition, subtraction, algebra, etc.) and work within those confines before you start to use these basic rules to push the boundaries and explore (proofs).
I suspect this has to with the fact that the bots were designed to be agents with no leadership. As you say, the bots performed very well in skirmishes, they lost because they didn't really know what to do beyond that.
They might fare better if one of the bots could act as a team captain and provide high level instructions to the team, i.e., assign people to harvest, defend, or push.
It seems like that's what the human teams have over the bots -- somebody telling the individual team members what to do.
OpenAI 5 had a number of glaring flaws in what it learned. It seemed to have never come to the conclusion that certain characters would have better outcomes if they focused on certain activities. Every character farmed, any character would grab the Aegis (reward item) after killing Roshan (usually this is reserved for a character likely to be in the middle of the fray, dealing a lot of damage).
Another problem was that, until the last few weeks before its matches against pro players, they had allowed both teams to have 5 couriers. This allowed OpenAI 5 to keep up relentless pressure by constantly bringing themselves consumable health regeneration items. This was so unlike a regular game of Dota that the community (including their opponents) complained because normally the courier's time is a valuable resource, and the courier can normally be killed if it is used too close to enemies. With an endless supply of healing items with no risk involved, it didn't even resemble a game of Dota. They did away with this for TI, and it revealed a serious weakness in their relentless-aggression strategy.
I get the impression most of OpenAI's games vs itself ended very quickly, with one team making a mistake during the relentless-aggression early game pushes. As a result, it seemed to have no idea what to do differently as the game went on.
I'll own up to wanting to chalk this up to "AI wisdom." The feed-the-carry strategy has always seemed like precisely the same kind of premature optimization that makes AI often easy to beat.
AI has long ways to go before it can defeat humans in a complex game like Dota2.
https://www.youtube.com/watch?v=IKVFZ28ybQs
That's the "gloves come off" AI. Granted, its a problem solved by raw, mechanical button-pushes at a rate beyond what is possible on a keyboard/mouse, but it demonstrates the greatest skills an AI or computer has.
Of course, no one wants to see an AI beat a human using known methodologies. They want to see an AI win at the abstract game of "a game of strategy".
From this perspective, I think the poker AI results over the past year or two have been far more interesting. AIs figuring out the Nash Equalibrium and playing bluffing games around it.
Sure, its a bit mechanical and mathematically inclined, but generalizing and solving the bluffing game is IMO far more useful.
https://github.com/cshenton/atari-leaderboard
And that list is incomplete if you want to include human high scores on non-emulators (not included in that list). This is even after the reinforcement learning algorithms have been given orders of magnitude more training time than humans. Furthermore, much of the machine performance over humans can be attributed to better reaction time.
IIRC, you can easily build a super-human Atari bot. Just add some off line planning (e.g. paper by Guo), or add manual rewards or features.
No one is doing since there is no point. We want to study Atari WITHOUT these "cheats", so that we can then apply these algos in more complex situations.
For generalized reinforcement learning algorithms. Humans almost always beat the "machine". The only cases where machine wins is on 100% hand eye coordination and duration.
All ML algorithms are terrible at figuring out how to plan de-novo. So anything that requires multiple contexts or planning is a fail.
Yet on the other hand, it is apparent that reinforcement learning algorithms have surpassed humans in board games such as Go and Chess.
Fair or not fair, there is no API to the real world. If we are going to create machines that can learn and think outside of simulations, however complex they may be, these machines will need a way to interact with the physical environment.
Gary Marcus is on the right track here. Building systems that can "handle the complexity of the real world", as per OpenAI's stated goal according to the article, is incredibly ambitious and pointing out the vast distance separating the current state-of-the-art from that lofty goal is, well, only fair.
I mean, if you think about it, back in the '70s, in the original AI Winter, one of the big criticisms of AI research was that it languished in simulated environments like blocks world and didn't perform nearly as well in the real world. And here we are today, celebrating a bright step on the path to conquering yet another simulation.
As a result, training in simulated environments doesn't help handle the full complexity of the real world. Even if we had robot bodies that could move as freely and manipulate objects with as much dexterity as they can do in simulations.
But you are right that it will fail on anything that is hard to simulate. And a lot of trivial things are very hard to simulate.
Tesla recently failed at things like plugging two cables together. Picking stuff like bags at Amazon warehouses may be another example.
> At least one previously undiscovered game mechanic, which allows players to recharge a certain weapon quickly by staying out of range of the enemy, has been discovered by the bots and passed on to humans
I assume they're talking about blink dagger, and something more advanced than "don't take damage"?
it was discovered that if you stay out of vision and cast raze, the other player does not get stick charges. that was the 1v1 shadow fiend bot a year ago though.
"Sumail pointed out that the bot had learned to cast razes out of the enemy’s vision. This was due to a mechanic we hadn’t known about: abilities cast outside of the enemy’s vision prevent the enemy from gaining a wand charge."
That's not an undiscovered mechanic in the world of Dota - that's been known for a while and at least documented since 2015 https://dota2.gamepedia.com/index.php?title=Magic_Stick&oldi....
I wouldn't be surprised if it was known before then, I certainly remember this from a while back.
Again, if that's not what it was, then I take back what I said, but if it was, I do think that statement is misleading as written.
Could we please lay this empty platitude to reast? "Planes don't fly by flapping their wings like birds, so why should computers think like humans?".
Well, except that we were only able to built flying machines (rather than floating ones) when we figured out why birds can fly [1].
And then, the whole of our computer science is based on the idea that a computer is, actually, the same kind of system as a human mind- a computational device, a machine that can compute everything and anything that can be computed. This is the deep insight that informs the ambition to create artificial intelligence: that brains are a kind of computer, computers are a kind of brain, and they can both compute the same kind of program, in other words- intelligence.
Though we may not know how human minds work exactly and therefore we can't readily copy them, it is thanks to Turing's and Church's insight that we even have computers today. And so, in a very real sense, we can only ever create thinking machines that do intelligence in the same way that humans do intelligence [2].
_______________________
[1] I mean, a boomerang is really an airfoil so I guess Australian Aborigines had figured out flight long before the Wright brothers, but I gues they didn't need flying vehicles back then?
[2] Unless there is a paradigm shift and we figure out a better way to do it etc etc disclaimer disclaimer.
Formal logic and the Turing machine were the model for computers.
Turing and Church did not invent computers based on their careful study of human biology and physiology.
Obviously, the Turing-Church thesis didn't have anything to do with brains. It does, however, have very important implications about our ambition to create human-like artificial minds.
Also, I'm sure that theoretically optimal "intelligence program" on silicon processors is vastly different from organic ones. They may share a lot, but the constraints are just so vastly different including the most important ones like energy efficiency or space availability.
That's not impossible! For example, it's easy to find algorithms that a computer can carry out without error that a human mind would really struggle with. Although this is a case of computational resources, rather than the expressive power of the computational apparatus, it's still the case that it's not always possible to find programs that both humans and computers can compute in practice.
So it may even be that, while computers can run some programs very efficiently, that humans can't, it's the other way around also and computers can't efficiently run the programs that humans can.
In which case of course, either the entire AI enterprise is doomed to failure, or we get lucky and there is some other way to do intelligence that is available to computers but not humans. Who knows!
Our current computers function almost nothing like human brains.
All we can say, is that we can write some programs now that give similar output to humans for certain well defined, limited problems. But even then, the mechanism via which those outputs are produced are very, very different.
The mistake is assuming machines that can perform tasks better than humans, will be essentially "like" humans in some fundamental, profound way. There is no reason to believe this is true. These machines may be able to mimic us very well, but their true underlying nature, their goals, ambitions, subjective experience, and values could be very, very different from us in ways that could be very unpleasant for us.
With respect, but I never said anything about computers functioning like human brains. What I said is that computers and minds can compute the same class of programs. I make no assumptions about underlying function.
>> The mistake is assuming machines that can perform tasks better than humans, will be essentially "like" humans in some fundamental, profound way.
I never stated any similar assumption either.
I think you're responding to some opinion that I didn't express.
> [1] I mean, a boomerang is really an airfoil so I guess Australian Aborigines had figured out flight long before the Wright brothers, but I guess they didn't need flying vehicles back then?
What? I'm not sure anyone would count a boomerang as "figuring out" flight let alone how birds fly.
I'm no historian but I'm pretty sure flight was first figured out mechanically through trial and error well before we had a good theoretical model for flight, let alone figuring out how birds fly. Either that or we have drastically different meaning for what it means to figure something out.
It took ages for us to fully understand how flight worked, well after we could fly. That's the basis of the apocryphal story:
https://www.snopes.com/fact-check/bumblebees-cant-fly/ https://rationalwiki.org/wiki/Bumblebee_argument
Oh, yes, indeed, there are limitations. The robot hand in question can only manipulate cubes and then only a specific kind of cube with standard dimensions, as far as it's possible to tell from all the demonstrations publicised by OpenAI. And, I'm guessing, if they had a robot hand able to play the yo-yo, they wouldn't hesitate to show it.
The bots excelled at things you expected computers to be good at, consistently quick reactions/mechanical skill and coordinated skirmishes. The bots fell flat on strategy, optimal resource usage (short and long term), and most importantly, coordinated decision-making.
So as an result, they dominated the early game which has always been heavily mechanics favored and absolutely fell apart mid/late-game when decision-making mattered (and often made decisions that benefited early game economy at the cost of mid/late game).
Computers being good at mechanics is not an impressive result.
With all due respect, the professionals who have played against the bot have been pretty impressed with previous iteration. The major roadblocks at TI were probably that the bot did not efficiently learn courier mechanics and that the draft was not done by the bot.
A human has ~200ms simple reaction time, then begins to move their fingers. Which then moves a mouse to target an area, which then requires hotkeys to be pushed. etc. etc.
Its still definitely a mechanical advantage, and it seems evident in the gameplay videos. The bots still seem "super-human" in their aim and some of the timing of spells.
What was interesting to me was that since the bots trained against themselves, they discarded this combo as being ineffective - the result being that they did not use this combo even against vulnerable human players.
This means that Dota 2 was balanced around the expectation that human's best reaction would be at least 400ms, because I've never seen Axe's blink calls being dodged unless the player had vision of Axe before the blink and thus was expecting it.
Multiple invincible couriers makes the game very similar to the 1v1 show match last year where the strategy of just out-live the opponent wins. The casters have never played that version of the game before, and wouldn't know to use that strategy.
The more interesting part was, the bots attempted to use that very same strategy in the single-vulnerable normal version of the game which was drastically less effective; you could only ferry out consumables at 1/5th the rate as before and the courier could be killed so it wasn't a riskless strategy like it was before.
I 100% disagree that the invincible couriers make the game "very similar" to the 1v1. There's many more layers of engineering that went into that version. The bots had to learn drafting, cooperation, highground pushing, Roshan, vision, invisibility, etc.
I'm amazed you can trivialize all this as a 'joke'.
In my opinion, what i see is a very good player who knows how to chain stun precisely without any strategic depth. If you claim to have built an AI system, which you ultimately want it to evolve to AGI, you at least expect some sort of strategic decision making at the macro level. Though since it has almost perfect micro, it can easily outweight the most of teams. So yeah, with that expectation I see this as a joke, too.
P.S. The model is trained with 128k cpus and 256 gpu. It is able to play 180 years worth of game in a day. Think about it.
It's the first line of the article: https://blog.openai.com/openai-five/
>Our team of five neural networks,
They use a hyperparameter called team spirit to cooperate. I don't think the goal of this is AGI at all, so I don't see why people are making that leap. But sure, for the geniuses of HN this must clearly be trivial.
I think they should have trained with 1 courier right from the beginning.
If I had to guess, they started out with the 5-courier setup because otherwise the bots would end up fighting over the courier -- just like a typical pub team. :)
The way I would guess this played out was that OpenAI tried multiple variants of the game, and this version ended up being good enough to beat human players. Of course, they later learnt that the bots were winning not because they were good, but the variant was imbalanced and its strategies unfamiliar to humans.
Deathball strategy has been a thing since even before 2014...
You have argued yourself into a corner. It happens.
No. We had implemented scripted courier logic for 1v1, and when switching to 5v5 the easiest starting point was to run five copies of that logic. (Dota's Turbo mode also has five invulnerable couriers.) In June, we had more important restrictions to remove — such as wards & Roshan; you personally were focused on the particular heroes we'd chosen: https://news.ycombinator.com/item?id=17392455. The couriers only become our most important restriction in August.
Of the list of skills you mentioned that the bots "learned", on the main stage, the draft was provided by humans, and from a significantly limited set, the bots never pressured highground, checked for Roshan when it was impossible for it to be up, had nonsensical placement of vision (placed vision where vision already existed), continuously invested in anti-invisibility consumables even when the opposition team had no invisibility.
If the mechanical skills were toned down to human levels, I don't think the latest iteration of the bot on the latest ruleset can even compete with the average human player, much less the pros.
The games they won weren't actual dota games - they were a simplified subset of dota that overweights teamfights and mechanical skill - exactly the subset that we'd expect an AI to be best at.
It's argued that drafting is one of the major components of competitive play and more often than not, draft advantage wins games, but that wasn't a factor in this match.
Only at human speeds.
Once you break the 10,000 APM barrier however, Zerglings start to dodge-tanks and other shenanigans begin to happen.
https://www.youtube.com/watch?v=IKVFZ28ybQs
Micro has a general advantage of growing at a rate of N^2. That is, 10 Zealots, with perfect micro, can perform roughly as well as 100 Zealots with the worst micro possible.
Ex: 100 Zealots come in one-at-a-time vs the 10-zealots in an inverted-V shape. The 10-zealots will defeat roughly 100.
The bigger and more complicated the board gets, the more and more micro becomes favored.
The announcers/casters were a bit ambiguous about this, AFAIK they said something like : "the teams agreed on a predefined, balanced list of heroes before the game...", and there was no drafting.
(I'm not native speaker, and I don't know much about Dota2 rules)