Interesting announcement timing at 10:30 PM PST on a Sunday. :P
The list of third-party gaming partners is extremely impressive, and a Docker config helps resolve the dependency hell that some of the AI packages require.
We've been pushing hard on some parts of Docker, and it's working pretty well. For example, reconfiguring iptables depending on what game you ask for. And it works fine to test things like that on my MacBook and then deploy to Kubernetes. Amazing.
That paper was 10 months ago. There have been many RL papers in the meantime, but sparsity is only a problem with respect to reward, not state or action, from what I can see.
Finally there is a trend of using recurrent neural network as a top component of the Q-network. Perhaps we will see even more sophisticated RNNs like DNC and Recurrent Entity Networks applied here.
Also we'll see meta-reinforcement learning applied to a curriculum of environments.
This is a bit out there, but it would be fun if OpenAI can get one of the mega popular multiplayer games under this (WoW, League of Legends, DOTA etc.).
Imagine an AI team in League of Legends world championship!
WoW I would be very surprised by. But all the MOBAs and esport FPSes and the like are fair game.
Mostly because, with the latter, improved AI means better competition and a deeper understanding (which boosts sales). With the former, improved AI means improved automation which means imbalanced economies.
I can't recall where, but I read that Tesla or Google were actually using GTA to train their self-driving cars, because it is a spectacularly advanced simulation of driving through an urban environment, so they didn't have to build their own.
There was an interesting academic research paper that showed you could train in GTA and transfer over to the KITTI dataset and do ok: https://arxiv.org/abs/1610.01983
deepdrive.io creator here - I'm actually not affiliated with the Berkeley project of the same name. There's also a DeepDriving at Princeton plus plenty of other (mostly perception) projects using GTAV, so it can be confusing. I'm hoping the GTAV for self-driving car efforts can start to standardize around the Universe integration though. Having worked on it, I can say firsthand that the Universe architecture is definitely amenable to sending radar, lidar, controlling the camera, bounding boxes, segmentation, and other types of info that the various sub-fields of self-driving are interested in. Super-excited to see how people use it!
There are hardcore boardgames you will find difficult to find human players willing to play with you. Campaign for North Africa takes 8-10 players and has an estimated playing time of 1000 hours [1]. An excerpt of a review written for this game:
> Are you a logistics major? Are you masochistic? Do you think that the calculations required to play a game should take longer than actually moving the units? Then do I have a game for you! Get yourself a copy of The Campaign for North Africa, and say goodbye to the family for a couple of months, if not years.
The Campaign for North Africa is the most detailed game that I have ever played. It isnt necessarily the most complicated, but for sheer size of the detail and planning involved, it is by far the most laborious and detail-oriented game that has ever been produced. As a first example, this is the only game that I know of that differentiates between British and German jerry cans for fuel. More about this later on.
The Campaign for North Africa is Richard Berg and SPIs simulation of the war in North Africa in the Second World War. The seven foot long mapsheet (divided into five sections), two sets of rulebooks, charts and tables galore and, oh yes, thousands of counters complete the game in a nice sturdy box, not the usual SPI flat game holder that falls apart. Most of this is standard SPI fare, with the functional but not pretty counters, standard three column style SPI rulebooks, and a fairly attractive map that does an excellent job of creating an epic sense of scale. True, this is the desert, and most of it is desolate, but the numerous tracks and roads, the coastal plains and mountains, and the railroad (both already built and railroad you can build as the game goes on) all combine to present an appealing picture of the area.
Each turn is one week of time, and each turn is broken down several stages. There is an initiative determination, naval convoy stage, stores expenditure stage, and then three operations stages. The Ops Stages are where most of the activity occurs. There are also stages that are used in the air game. I did not play the Air Game for the purpose of this review, but did play with the advanced logistics.
The game also includes on of each type of chart, which can be used to make copies. I made my own in Excel. There are charts for Division and Brigade organization, truck convoy sheets, naval convoy sheets, prisoner sheets, broken down and destroyed vehicle sheets, supply dump sheets, sheets for the air game and more. I even created a couple of my own for production and independent units. As each Division in the game needs its own Org chart, which fit best on legal size paper, these are a lot of charts and sheets to keep track of. All of these must be filled out before the game even starts, and just setting up for the beginning of the game requires filling out hours (literally) of paperwork. And for heavens sake, dont use pen! Much of what you write in the charts at the beginning of the game will be erased by the end of the first turn. After every movement, every combat, just sitting there and doing nothing will require updating of the org charts for every unit in the game.
Tabletop simulator would making an interesting training environment. I may not be able to train a good go player, but a table flipping sore loser is probably doable.
for some reason, the idea of autonomous bots crawling around the internet also unsettles me. i guess it really depends on what kind of rewards you train it for.
Nice video, but the jump from solving super simple 2d games, by feedback of binary win/lose conditions, to solving tasks in 3d open world simulations will require an un-imaginably gigantic leap in processing and knowledge. Additionally neural nets have already shown they are not sufficiently good enough at generalizing, and only work well at the specific tasks they were trained for. So the idea that an AI that can play GTA would also be able to 'solve' climate change is odd.
I'm far from an expert, but I thought the poor generalized performance of neural nets was largely associated with the complexity of the network (number of neurons, etc), and the training data.
Is there something more specific about the application of neural nets to generalized problems that makes them unsuitable?
With this platform (and Gym) it seems like a large part of their strategy for "democratizing AI" is to grow the amateur research community. By making it easier for an individual to play around and conduct experiments, they are hoping enable progress to emerge from anywhere instead of just from wealthy companies and elite universities.
It is also a great way to be able to track and organize what is being created rather than having to sort through amateur projects scattered across the web or research publications that often lack accompanying code.
Edit:
Some key ways they're making it easier for amateurs:
* Starting point for problems to solve
* Way to get noticed (instead of needing a university/company brand)
* Technological infrastructure for building and testing. The diversity of tools they brought together to build this platform is very impressive.
This is amazing! I was thinking of this problem when I saw a friend making a stop-motion video. The steps are super repetitive and I asked him, "maybe an DeepMind Atari-style RL agent can learn how to do this?" But I didn't want to do what DeepMind did to emulate Atari games with an Adobe editing tool. This is an experiment that I can now run.
Never saw how stop-motion videos are edited. What's so repetitive? I thought that once you've taken all your pictures, you just put them sequentially and removed any frames that seemed off. Maybe you also need to decide how much time to leave every frame?
This is true, but some are more complex, which is also due to the choice of tool. For example, [1], made by Chris King, using Adobe Premiere. This is a time-lapse of the process that he used to make parts of [2]. Notice the pattern that emerges when sequential images start lining up.
It depends how many hyperparameters there are. Many popular general-purpose ML algorithms have only a handful of numbers for hyperparameters, so they don't embody much human input. And they can sometimes be tuned automatically.
Also, an algorithms that can learn 100s of different games with the same hyperparameters is more highly regarded than one that needs different hyperparameters for each.
If requests are being taken, it would be useful to be able to search through the listed environments. And a poker environ for the internet section would be a good balance of fun, widely appreciable and a straight forward but very non-trivial environment.
If it's true, I believe we have to wait for the OpenAI team to build new gym environments before we can train in new games.
I only briefly poked around because it's nearing on midnight here - maybe you can pull open the examples included and work out how to rewire them to work on new games, maybe not. Either way, I've got a particular use case I'd like to make a gym for so I'm interested in finding out.
It looks like the image from the server and control information to the server is sent through the VNC protocol. Other information such as the reward signal from the environment server is sent through a WebSockets protocol using JSON:
You should be able to implement this protocol for your environment and run a VNC server for the rest. A new class for the client representing your environment can be based on this:
Related but slightly off-topic, there is a great sci-fi story by Ted Chiang (the same author who made the story behind Arrival film) about humans raising AIs in an artificial world. The premise is that if we want AIs to act like humans, we must teach them like we teach humans: http://subterraneanpress.com/magazine/fall_2010/fiction_the_...
The people who will end up raising AI's will not want them to act like humans. You already see it in their current uses. They create them to maximize profit. So, in a way, their owners (corporations) have created them in their image.
At Asteria we're using a Agent-System-Interface model, and are building models around observations of your own activity.
I 100% agree we need to teach them like humans, so they at least can build a model of how humans interact and participate with one another. At the least this will teach them about us, more than it will teach them about anything else. And if we want to participate and collaborate with that future of AI we need to have these models.
I'd love to see AI, using games, master the art of determining a depth for objects in the scene. If you ask a person, "about how far away is that car?", they often give you an okay answer that is at least in the same magnitude as the actual distance 1 m, 10 m, 100 m, 1000 m. If AI could do that, you could then navigate an environment in the real world better using only a camera or two. So you start with a virtual world that looks real, train up the bot, then use it to navigate in the real world. Has this already been accomplished?
More generally, imagine AI that could learn the physics of the world. For example, if the ball is rolling away, the AI should be able to predict that the ball will look smaller on the next frame.
Going further, if the ball is about to roll under a shadow, the AI should predict that the ball will become a darker shade of green.
(After several years working in a robotics research company, these kinds of capabilities are exactly what we determined would be necessary for robot AI.)
Agree, it's not easy. Learning the basics, for example projecting a rectangle with 3d coordinates to 2d coordinates, then feeding the 2d coordinates into a NN and ask for the (depth) third dimension. Can you teach the NN a perspective transform? Can you rotate the rectangle and recognize rotation. Can you add other rectangles to the scene and detect each? Can you add color and lighting to infer more properties and get better results? Shine some more info on the problem ;)
These are like unit tests of AI (basic shapes and transforms) and I agree physical reckoning is at the top, one of the big tests that is a capstone and something beautiful to behold in nature (eg. sports). Maybe the a virtual soccer game at the end?
From my lidar experience, I wanted to reach for a model rather than deal with noisy sensor data. I want to generate the output (3d world) with my model, then the NN learns the inverse (eg. the scene graph used to generate the scene).
I enjoy thinking about this stuff, though it really makes my head spiral sometimes when I relate it to my own reality. It's easy to feel like you're losing touch.
138 comments
[ 6.0 ms ] story [ 200 ms ] threadThe list of third-party gaming partners is extremely impressive, and a Docker config helps resolve the dependency hell that some of the AI packages require.
https://universe.openai.com/envs#pc_games
https://arxiv.org/abs/1602.01783
Is there a way to deal with "sparse" training data (state, action, reward) triples -- sparse in "state"?
Finally there is a trend of using recurrent neural network as a top component of the Q-network. Perhaps we will see even more sophisticated RNNs like DNC and Recurrent Entity Networks applied here. Also we'll see meta-reinforcement learning applied to a curriculum of environments.
Imagine an AI team in League of Legends world championship!
https://deepmind.com/blog/deepmind-and-blizzard-release-star...
Mostly because, with the latter, improved AI means better competition and a deeper understanding (which boosts sales). With the former, improved AI means improved automation which means imbalanced economies.
gran tourismo is advanced, for a videogame at least
> Are you a logistics major? Are you masochistic? Do you think that the calculations required to play a game should take longer than actually moving the units? Then do I have a game for you! Get yourself a copy of The Campaign for North Africa, and say goodbye to the family for a couple of months, if not years.
The Campaign for North Africa is the most detailed game that I have ever played. It isnt necessarily the most complicated, but for sheer size of the detail and planning involved, it is by far the most laborious and detail-oriented game that has ever been produced. As a first example, this is the only game that I know of that differentiates between British and German jerry cans for fuel. More about this later on.
The Campaign for North Africa is Richard Berg and SPIs simulation of the war in North Africa in the Second World War. The seven foot long mapsheet (divided into five sections), two sets of rulebooks, charts and tables galore and, oh yes, thousands of counters complete the game in a nice sturdy box, not the usual SPI flat game holder that falls apart. Most of this is standard SPI fare, with the functional but not pretty counters, standard three column style SPI rulebooks, and a fairly attractive map that does an excellent job of creating an epic sense of scale. True, this is the desert, and most of it is desolate, but the numerous tracks and roads, the coastal plains and mountains, and the railroad (both already built and railroad you can build as the game goes on) all combine to present an appealing picture of the area.
Each turn is one week of time, and each turn is broken down several stages. There is an initiative determination, naval convoy stage, stores expenditure stage, and then three operations stages. The Ops Stages are where most of the activity occurs. There are also stages that are used in the air game. I did not play the Air Game for the purpose of this review, but did play with the advanced logistics.
The game also includes on of each type of chart, which can be used to make copies. I made my own in Excel. There are charts for Division and Brigade organization, truck convoy sheets, naval convoy sheets, prisoner sheets, broken down and destroyed vehicle sheets, supply dump sheets, sheets for the air game and more. I even created a couple of my own for production and independent units. As each Division in the game needs its own Org chart, which fit best on legal size paper, these are a lot of charts and sheets to keep track of. All of these must be filled out before the game even starts, and just setting up for the beginning of the game requires filling out hours (literally) of paperwork. And for heavens sake, dont use pen! Much of what you write in the charts at the beginning of the game will be erased by the end of the first turn. After every movement, every combat, just sitting there and doing nothing will require updating of the org charts for every unit in the game.
[1] https://boardgamegeek.com/boardgame/4815/campaign-north-afri...
What if AI can do anything what can human do you with a browser over the phone?
Also love "bring your own Docker container format".
https://www.youtube.com/watch?v=mGYU5t8MO7s
Is there something more specific about the application of neural nets to generalized problems that makes them unsuitable?
You and your personality are fine, but the jump between the intro and the payoff is jarring.
You need to hold our hand a little more ... jus' sayin'.
It is also a great way to be able to track and organize what is being created rather than having to sort through amateur projects scattered across the web or research publications that often lack accompanying code.
Edit:
Some key ways they're making it easier for amateurs:
* Starting point for problems to solve
* Way to get noticed (instead of needing a university/company brand)
* Technological infrastructure for building and testing. The diversity of tools they brought together to build this platform is very impressive.
[1] https://youtu.be/M7Hr83OI-rs
[2] https://youtu.be/1-rFV_d6RH8
Related to the blog post: https://openai.com/blog/universe/
Also, an algorithms that can learn 100s of different games with the same hyperparameters is more highly regarded than one that needs different hyperparameters for each.
If requests are being taken, it would be useful to be able to search through the listed environments. And a poker environ for the internet section would be a good balance of fun, widely appreciable and a straight forward but very non-trivial environment.
I only briefly poked around because it's nearing on midnight here - maybe you can pull open the examples included and work out how to rewire them to work on new games, maybe not. Either way, I've got a particular use case I'd like to make a gym for so I'm interested in finding out.
https://github.com/openai/universe/blob/master/doc/protocols...
You should be able to implement this protocol for your environment and run a VNC server for the rest. A new class for the client representing your environment can be based on this:
https://github.com/openai/universe/blob/master/universe/envs...
Then register the class with OpenAI Gym:
https://github.com/openai/universe/blob/master/universe/__in...
After creating the environment using gym.make you need to add information about your remote in the call to configure:
env = gym.make('gtav.SaneDriving-v0')
env.configure(remotes="vnc://localhost:vnc_port+rewarder_port")
https://github.com/openai/gym/blob/master/gym/core.py#L234
https://github.com/openai/universe/blob/master/universe/envs...
This is only based on a cursory reading, but it should be possible to use custom environments with OpenAI Universe as it is today.
I 100% agree we need to teach them like humans, so they at least can build a model of how humans interact and participate with one another. At the least this will teach them about us, more than it will teach them about anything else. And if we want to participate and collaborate with that future of AI we need to have these models.
Any applications with a keyboard and mouse? Can I use emacs and have it start learning to code?
More generally, imagine AI that could learn the physics of the world. For example, if the ball is rolling away, the AI should be able to predict that the ball will look smaller on the next frame.
Going further, if the ball is about to roll under a shadow, the AI should predict that the ball will become a darker shade of green.
(After several years working in a robotics research company, these kinds of capabilities are exactly what we determined would be necessary for robot AI.)
These are like unit tests of AI (basic shapes and transforms) and I agree physical reckoning is at the top, one of the big tests that is a capstone and something beautiful to behold in nature (eg. sports). Maybe the a virtual soccer game at the end?
From my lidar experience, I wanted to reach for a model rather than deal with noisy sensor data. I want to generate the output (3d world) with my model, then the NN learns the inverse (eg. the scene graph used to generate the scene).
I enjoy thinking about this stuff, though it really makes my head spiral sometimes when I relate it to my own reality. It's easy to feel like you're losing touch.