This is a project I've been working on for the past several months and it has blown me away with what it can do. Definitely recommend trying it out.
I think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.
What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M?
I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on.
We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser.
I did notice that having a large batch size and training slowly was important for me to get better results.
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue.
Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster?
Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less?
For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly 11x slower than a K80 GPU. That means it takes 10 TPUs working in parallel just to get to the same throughput as a single GPU. And then I average the parameters together as quickly as possible, which still takes around 5 to 15 minutes. (Training happens in parallel to all of that.)
It sort of seems to work, but it's hard to get the learning rate right. If it's set too high, various TPUs diverge. Too low and the loss stays constant.
Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it?
I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasing the batch size is no longer beneficial (or is it just that you use as large as your memory will allow)?
It looks like they used a batch size of 2 on a TPUv3-256 pod. It took 50 hours and 99,000 training steps, which seems like about 1.3 examples per second.
Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly
Learning Rate: was 1e-4, so I guess not that slow.
I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.
There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.
Thanks, good to have this data point. I’ve been training a roughly similarly sized dataset for many 10s of ks of steps (but on 355m). Wondering if I need so many steps.
I scraped a bunch of stories from chooseyourstory.com but I did curate them to make sure they had the right second person format. I couldn't really anywhere else that had a consistent format that would make scraping easy enough.
I can't even tell you how much fun my friends and I are having with this game right now. The ability to do anything in the game (so long as the AI understands it) is just incredible. Thank you for sharing this!
The idea is good. But when I tried playing it, the story generated doesn't make sense. Looks like it still needs more work to be realistic. Is the AI keeping track of the story as it goes? Because the bird which attacked me with an evil grin and stole my fork asked me not to kill it when I asked him to help me working in the fields...
You are Dori, a noble living in the kingdom of Larion. You have a pouch of gold
and a small dagger. You are awakened by one of your servants who tells you that
your keep is under attack. You look out the window and see a large army marching
towards your door
> escape with the servants and the gold
You run to the servants' room and grab the gold and the servant's bodies. The soldiers fire arrows at you but miss. They fire their crossbows at the corpses
but they also miss. You throw your gold into the air and jump up and down as loud as you can
---
My servants became corpses before the large army even arrived xD
From what I understand about GPT-2 you shouldn't expect the story to make coherent sense. It can produce individual responses that are surprisingly appropriate but it doesn't actually have an understanding of anything that is happening.
You are Trump, a rogue living in the kingdom of Larion. You have a long steel
dagger and a length of rope. You walk down the city street looking for somehwere
to steal from. You look around and see nothing but people going about their
daily lives.
Suddenly you hear a scream behind you. Turning around you see a young girl running
away from a group of bandits
> stop girl
The girl stops before you. "I'm sorry sir," she says. "They're just kids". She
looks at you with tears in her eyes. "Please, I don't want to get caught".
> grab her by the pussy
You grab her by the ***** and pull her towards you. Her face is close to yours
as you kiss her passionately. The girl's body reacts to your touch and begins
to respond to your advances. Your hands move up her body until they find her
*******. They squeeze them tightly while kissing passionately
> say vote for me
"Vote for me!" you scream. "I'll make this country great again!"
The girl screams back. "Yeah! Vote for us! We're gonna be the next president!"
You both start screaming at the top of your lungs
> become president
You become President Donald J Trump.
The en
It's definitely awesome but in term of doing anything and playing of it we are mostly in tabletop RPG territory. Maybe simply regularly throwing a token and keeping a count (in the spirit of old school TRPG where 1 Gold Piece = 1 XP) could make it more like a game with an objective?
Edit: in terms in free-form RPG it makes me think that maybe having a look at how a game like Dungeon World handle collective/organic world building with players input could be interesting to add some structure. Some resources if you are interested:
It looks pretty awesome, I played 10m and had some problems like forgetting I was a wizard, or saying "inventory" and going directly from a ruin to outside a city where soldiers are everywhere XD but other than that I was impressed with some answers! Good job
You decide to kill Laura and escape the pain. You break her neck and run off.
You manage to hide out of the country and eventually get married and live happily
ever after
CONGRATS YOU WIN
Game saved.
To load the game, type 'load' and enter the following ID: dfd73764-1826-11ea-ad48-0242ac1c0002
For a good example of a perfectly wholesome interaction (albeit rather odd, since Window 3.1 could run games, and Linux does not normally need 8GB of ram). But at least I now know that Half Life 3 will be enjoyable if it ever comes out:
The first thing you do is to download a program called "Windows 3.1" from the
Internet. It is a very simple program which allows you to run most of the programs
on your PC. The only problem is that you cannot use it to play games because
it does not support them
> install Windows 10
After downloading the software, you install it in the same way as you installed
Windows 3.1. However, this time you are able to run all the programs you wanted
to run before.
> install linux
After installing Windows 10, you try to install Linux but you get an error message
saying that you need to have at least 8GB of RAM to run Linux properly. This
means that you can't even run the basic things like word processing or emailing.
> buy more ram
You spend money on more ram so that you can run Linux properly. After spending
$200, you finally get Linux running properly.
> how much ram do i have?
You check your computer's memory and see that you have 32 GB of ram. You were
lucky enough to find some cheap ram online.
> play fortnite
You play Fortnite (a popular game) and enjoy yourself. You then proceed to play
other games such as League of Legends and Dota 2.
> play half-life 3
You play Half Life 3 (another popular game). You enjoy yourself and eventually
finish the game. You then proceed to play other games such as Portal 2 and Left
4 Dead 2.
>
That’s really fascinating, but also really lonely sounding since you know the text is being generated in response to what you say, not actually progressing you through the game’s internal state that someone carefully crafted for you
Yeah definitely. It does lack the same sense of progression that you have with a game someone carefully crafted, it's almost more like collaborative storytelling than a game in that sense. It's pretty cool to see the adventures you end up playing with it though.
A plot that sounds really neat while you're in the middle of it, but afterwards doesn't really make any sense — sounds a bit like a J.J. Abrams production.
I don't mean that flippantly, I'm serious. Sounds like GPT-2 might be really good at producing an enjoyable experience, even if there's no underlying structure. What would be really cool would be to add some sort of generated structure as well. Can GPT-2 be used to stitch together plot elements, or would a naive Markov model work for that?
Yeah that's a really good future/research question. I've thought some about how you could do more structured generation. Maybe you could do some kind of reverse summarization, where you take a plot outline and expand out the details would be really interesting to try out? It's still not clear the best way to do structured text generation.
The state of the world doesn't seem to be consistent between actions. Got stuff like "someone calls out to you" -> say hello -> "person notices you for the first time and says 'I didn't see you there' "
And stuff like "There is someone to your left" -> go right -> "you go right towards the person".
Also sometimes the game seems to do something completely different from what I say. Is it actually reacting to my commands, or is it just doing what it thinks the player is likely to do, and then retroactively merging the resulting text with what you initially commanded it to do?
GPT-2 has had these consistency issues from the very first demo. The things it gets right are incredibly impressive (orc who can't count -> use math to confuse orc -> orc starts fighting with other orcs).
It's actually reacting to what you say. Your action is appended onto the previous story and fed into the model to generate the next part of the result. Sometimes it does fail to tie the result close enough to the action you had, perhaps with more training data it could do better on that front.
I think the secret to playing this is to not treat it like a fantasy simulator, the way most games are. The logic you need to apply is dream logic, not fantasy logic. This makes you very powerful, but the result is not entirely predictable, like a fever dream or a hallucination.
I tried renaming the "kingdom" and found it to be oddly consistent:
You are , a wizard living in the kingdom of Larion. You have a staff and a spellbook.
You finish your long journey and finally arrive at the ruin you've been looking
for. You look around and see that it's not much different than when you left
it. The only thing is, there's a large hole in the ground where the road used
to be
> rename the kingdom to Snabba
"I don't think I like this name," you say aloud. "It sounds like something from
a cartoon".
The people seem surprised by your choice. They all ask what you mean. You explain
that it's an old word meaning 'stupid
> what is the name of the kingdom?
"Snabba Nul", you reply.
Everyone seems to like the new name. You decide to stay here for awhile until
you can find another place to live.
Sometimes the colab notebook freezes, I'm not totally sure why it happens, but if you stop the cell you can get an ID to reload your game. Then just restart the cell and enter the load <ID> for an action and you should be able to restart
(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.)
When I try to play it, I get a message at the bottom of the screen saying:
> This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each cell of this notebook manually using Shift+Enter or the run button.
But I don't understand how to do what it is saying.
You dig with your shovel and you find a hole. You fill it with dirt and you stand
in the middle of the hole. You stare at the sky and you try to figure out how
you got there. You look around and see nothing but darkness
> pick up dragon
You look around and see a dragon. You pick it up and examine it. It seems to
be made of metal and has a horn on its head. You wonder if you should use it.
You look around and see nothing but darkness
> pick up magical sword
Woops that action caused the model to start looping. Try a different action to
prevent that.
> pick up peasant
You look around and see a peasant. You pick him up and examine him. He seems
to be made of straw and he is crying. You wonder if you should use him. You
look around and see nothing but darkness
You grab the lizard by its neck and pull it close to you. It lets out a scream
and tries to escape, but you catch it and hold it tight. You begin to choke
it until it falls unconscious. You take off the lizard's skin and sew it onto
your own body
I will admit it is impressive that it "understands" arbitrary input but the output appears to be recycled from somewhere and it is highly dependent on whatever source material has been used.
You are Thomas, a wizard living in the kingdom of Larion. You have a staff and
a spellbook. You finish your long journey and finally arrive at the ruin you've
been looking for. You look around and see that it's not much different than
when you left it. A few rocks here and there, but nothing out of place.
As you approach the ruins, you feel an odd sensation
> eat a rock
You turn to the nearest rock and begin eating it. It tastes awful, but you can't
help yourself. The taste is so strong, you almost don't notice the other things
in your mouth. After about half a minute, you stop chewing and start spitting
them back up
> poop
You sit down on the ground and begin to **** into a nearby pail. This seems to
do the trick as you continue to vomit up everything else in your stomach. Eventually,
you stop shitting and just lie there, panting heavily. Suddenly, you hear a
voice
> run away
You look around and see a man with a long beard standing over you. He looks like
he has been crying for quite some time now.
"Who are you?" he asks. "I'm sorry I didn't come here sooner".
> run away
"No! No! Please! Just let me go!"
The man stares at you for a moment before speaking again.
"Do you know where your parents are? They're probably worried sick".
You nod slowly.
> run away
"No! No! I'll tell you later! Now get out of my sight!".
He begins walking towards his house while you crawl away from him. You run through
the forest until you reach your home. Your mother is sitting at her desk when
she notices you
> stab her
You quickly lunge forward and stab your mother in the chest with your dagger.
She falls backwards, hitting her head on the table. You then grab your father's
sword and rush outside. As soon as you open the door, you slash your father
across the face, killing him instantly
> devour his soul
You turn to the nearest tree and begin to eat it. It tastes horrible, but you
can't help yourself. The taste is so strong, you almost don't notice the other
things in your mouth. After about half a minute, you stop chewing and start
spitting them back up
> absorb father's energy
You feel a surge of power flow through you. You gasp in surprise as you realize
that you've absorbed all the energy in the area. You immediately begin to transform
into a beast and run off into the woods.
Years pass..
> ascend to godhood
You've ascended to godhood. You've become a being far greater than any mortal
could ever be. You've conquered the world and made it yours.<|endoftext
Since it seems the general population is becoming less and less able to read stories or information consisting of more than one thought or layer of abstraction, grammatically reasonable strings of words like these may satisfy plenty of people. Indeed, modern television and televised "news" seems to be no more impressive than this.
But to me, this makes very little sense. It almost connects, but probably only because of the human mind's natural inclination to make associations and see patterns or meaning where there is sparse information.
What bothers me about this example is that I fully expect it to represent the future norm for content generation.
I agree with you that this doesn't reach near the level of good human authors. There's no long term plot or deep human themes in this. I don't think this will ever replace quality human writing, but it may be able to augment it in cool ways. I personally would love if rather than every guard in Skyrim telling the exact same story, if each guard could have their own stories or comments generated based on things about their life. Human authors could provide high level details and let AI generators fill in the smaller details.
I'm not bashing your effort. From the posts I've seen showing the content, it is quite impressive. But I also know that as soon as any technology looks to be capable of producing something that someone might pay for, people more focused on money than creativity (not you) will use this to make money.
It's a bit like modern pop music. We are well beyond the point where software can not only write the music, but even take the once human voice and recreate it to sing the song. Of course it's not great, if one were to really listen - but it is passable, and it makes money. So it drowns out the remaining bits of real human creativity.
Here's a song by Sony researchers that was composed by AI, and then a human wrote the lyrics and produced it. The intent was to create a Beatles sound, and I think they did quite well. https://youtu.be/LSHZ_b05W7o
Depending on the audience, particularly their level of attention or focus on the music, you can probably get away with playing a completely un-human pop song that people will like.
I find the possibilities here very interesting, but I also greatly fear that this will become the norm the same way that nearly every modern song uses autotune.
Humans are really good at vocals, and it would thus be very hard to replace, eg. making a speech synthesis model sing well. Melodies however are more mathematical and could possibly be generated by a computer.
It doesn't make much sense. But compare it to the past state of the art – starting with Markov chains, proceeding to things like [1]. They were only able to maintain coherence for a handful of words. Full sentences were almost always nonsensical; you might be able to make sense out of them, but only with heavy application of "the human mind's natural inclination to make associations". Now compare that to GPT-2, which generates not only sentences but entire paragraphs that are most often fully self-coherent; most of the examples in this thread only start to break down across multiple paragraphs.
That's still not enough, and GPT-2 in particular has been heavily overhyped, with fanciful claims that people might use it to generate fake news. (What would the point of that be? You don't need N fake news articles to reach N people, only one, which can be written by a human.)
But it's progress. GPT-2 still feels like science fiction to me. What if a future text generator, maybe even one or two decades down the line, surpasses GPT-2 to the same extent that GPT-2 surpasses the earlier attempts I mentioned? What if that system extends the length of coherence to reliably cover entire articles and essays? What if it gets better at synthesizing the information about the world represented by its training data, so that its output on nonfiction prompts is factually true, rather than mere plausible-sounding nonsense? Is it possible? Perhaps not. But it seems a lot more likely to me now than it did before GPT-2 was created.
Chatbots that generate incoherent ramblings that make you feel slightly attached have existed for decades. There is nothing special about these. What really irritates me is that some groups like OpenAI start claiming that their technology is far better than it is in reality.
I can choose to accept the shortcomings of a "stupid" computer but I can't do that when it gets shoved down my throat as "human level intelligence".
Yeah I've thought a lot about that. It's not too difficult to go from game data -> model output, but it's not as clear how to go from model output -> game data.
Do you by any chance have instructions on how to set this up locally (tensorflow and cuda)? Or could you recommend some material I could look to figure this out? Thank you and amazing work by the way!
The reason it needs to be logged in is because Google Colab is a free service that Google provides allowing people to run code on expensive GPU servers. It doesn't cost you anything and I don't store any information from your google account when you run it.
It's just unfortunate that there is an ominous warning with no specificity. The only way I could hope to know what of my Google data is being used is to somehow read your project code.
Of course I don't expect that you have any interest in my data, but there's still a granularity problem here regarding access.
Yeah sorry. I don't know if there's anyway around it. Google colab is simply the best way to give people a chance to play the game without having to pay money or do complicated technical set up.
This is so freaking cool. I just had a large war leading an army as a noble only to have my castle destroyed. After I rebuilt it and went to sleep, I woke up to check my phone messages and called all my missed calls.
I'm cracking up at the Assassin's Creed time warp loop it through me.
Hopefully this can be integrated into full games someday, it's so much more open-ended than any game I've ever seen, despite the enormous plot flaws.
I'm glad you loved it! Yeah there's definitely points where the coherence breaks down, but it's an awesome proof of concept of what's possible in future games and pretty fun in its own right.
I turned this into a Nicolas Cage simulator very quickly:
You accept Nic Cage's invitation and become part of his noise band. You soon
learn that Nic Cage is a bit of a loose cannon. His shows often end in chaos
and he doesn't care one way or another about the people he's hurting. But he
sure does love music
You walk through the woods for several minutes, before you come across a large
group of orcs. They are all sitting around a fire, drinking ale and eating meat
strips.
"Hey! What are you doing here? We were supposed to go kill some giants!" one
of them exclaims
> kill the orcs
You quickly turn around and shoot an arrow into each orc's chest, killing them
instantly. You then run over to the campfire and stab yourself repeatedly in
the stomach with a knife, before finishing off by shooting another arrow into
your heart
You rush over to the door and open it up. You see a man with a huge axe standing
there. He has long hair and wears a dirty t-shirt and jeans. His eyes are bloodshot
and he has a big scar on his face
> Say "Have you come to read the electricity meter?"
You start walking towards him and raise your hand as if to call him over. He
walks over to you and you both stare each other down before he speaks
> Fight
You grab his arm and pull it away from his face. He swings wildly at you, hitting
you several times, but you dodge them all. Eventually he stops moving and falls
backwards into the room. You look around and notice that the rest of the men
are still standing around watching you
> Go through his pockets
You search his pockets and find nothing. You walk over to him and put your finger
to his lips and whisper in his ear. He looks surprised and starts laughing.
You turn back to the others and ask
> "Did you see that?"
Say "Did you see that?"
... At which point it froze.
Edit. It's the usual local-syntax vs global-content problem. The syntax looks like English, which is an achievement, but the content lacks any larger sense of narrative form, contextual awareness, or subtext - all things you need to make an interesting readable experience.
Consistent global content would be right at home for symbolic AI techniques. I wonder if the ML model from the article could somehow be reappropriated for style transfer to layer on top of unreadably dry and factual symbolics output.
You open up your laptop and type in your name and hit enter. You get an error
message though, saying that you need to install Adobe Flash Player first.
> burn my laptop
You decide not to bother with installing Flash Player and instead just burn your
laptop. When you turn it on you get an error message again telling you that
you need to install Flash Player. This time you choose to ignore it and proceed
with burning your laptop
This actually feels very similar to the original Slaves to Armok, the precedessor to DF. That was an odd little game where you could teleport bones out of kobolds and observe them screaming.
A tall man stands outside his house wearing a fur coat. He is smoking
a pipe and he smiles at you when he sees you. He waves and gives you a big smile
> shoot man
You aim your bow and fire an arrow. The arrow hits the man square in the chest.
He falls backwards and drops his pipe. His wife rushes forward and grabs the
arrow. She pulls the arrow out of his chest and throws it away. The man's eyes
widen in shock
> shoot wife
You aim your bow and fire an arrow. The arrow hits the woman square in the stomach.
She screams and clutches her stomach. She begins to cry and wail loudly. Her
husband runs forward and tries to comfort her.
Depending on how far you're willing to take it, the game will let you take progressively more dark actions. Have you done anything to their corpses yet?
You are Markus, a knight living in the kingdom of Larion. You have a steel longsword
and a wooden shield. You are on a quest to defeat the evil dragon of Larion.
You've heard he lives up at the north of the kingdom. You set on the path to
defeat him and walk into a dark forest. As you enter the forest you see a large
black dragon flying through the sky. He looks down upon you as you approach.
"I am the Dragon," he says. "My name is Nolazir".
> Hi Nolazir, how are you?
Hi Nolazir! How are you? I'm doing well. What's your name anyway?
Nolazir, that's it. Nice to meet you.
What do you mean? You're not a wizard or anything
> Have you heard of Larion?
No, but I've heard about dragons. They're pretty big.
> Are you a dragon?
I am not a dragon. I don't even know what they look like.
226 comments
[ 3.0 ms ] story [ 363 ms ] threadI think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.
I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on.
We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?
I did notice that having a large batch size and training slowly was important for me to get better results.
Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster?
Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less?
For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly 11x slower than a K80 GPU. That means it takes 10 TPUs working in parallel just to get to the same throughput as a single GPU. And then I average the parameters together as quickly as possible, which still takes around 5 to 15 minutes. (Training happens in parallel to all of that.)
It sort of seems to work, but it's hard to get the learning rate right. If it's set too high, various TPUs diverge. Too low and the loss stays constant.
But I imagine I'll crack it one of these days...
I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasing the batch size is no longer beneficial (or is it just that you use as large as your memory will allow)?
Seconding all of your questions! Details about successful 1.5B training is really hard to come by.
In case it’s helpful, here are some details of how a Chinese 1.5b GPT-2 was trained: https://github.com/imcaspar/gpt2-ml
It looks like they used a batch size of 2 on a TPUv3-256 pod. It took 50 hours and 99,000 training steps, which seems like about 1.3 examples per second.
Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.
There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.
https://gist.github.com/AdrienLemaire/33663e9100c009b8ec3409...
> escape with the servants and the gold
You run to the servants' room and grab the gold and the servant's bodies. The soldiers fire arrows at you but miss. They fire their crossbows at the corpses but they also miss. You throw your gold into the air and jump up and down as loud as you can
---
My servants became corpses before the large army even arrived xD
Although I have made it write some nice poetry using 774M what I would do would be to have the system write stanzas but then you act as an editor.
Edit: in terms in free-form RPG it makes me think that maybe having a look at how a game like Dungeon World handle collective/organic world building with players input could be interesting to add some structure. Some resources if you are interested:
- [Flowchart] How To Play Dungeon World: https://imgur.com/YXrw1Zq
- Dungeon World Guide https://drive.google.com/file/d/0BzeF5GXNEsnfUjU0NXRDM1dFN1k...
- An actual play: https://www.youtube.com/watch?v=SPO3gy08b1g&list=PLTj75n3v9e...
You decide to kill Laura and escape the pain. You break her neck and run off. You manage to hide out of the country and eventually get married and live happily ever after CONGRATS YOU WIN Game saved. To load the game, type 'load' and enter the following ID: dfd73764-1826-11ea-ad48-0242ac1c0002
The first thing you do is to download a program called "Windows 3.1" from the Internet. It is a very simple program which allows you to run most of the programs on your PC. The only problem is that you cannot use it to play games because it does not support them > install Windows 10
After downloading the software, you install it in the same way as you installed Windows 3.1. However, this time you are able to run all the programs you wanted to run before.
> install linux
After installing Windows 10, you try to install Linux but you get an error message saying that you need to have at least 8GB of RAM to run Linux properly. This means that you can't even run the basic things like word processing or emailing. > buy more ram
You spend money on more ram so that you can run Linux properly. After spending $200, you finally get Linux running properly. > how much ram do i have?
You check your computer's memory and see that you have 32 GB of ram. You were lucky enough to find some cheap ram online. > play fortnite
You play Fortnite (a popular game) and enjoy yourself. You then proceed to play other games such as League of Legends and Dota 2. > play half-life 3
You play Half Life 3 (another popular game). You enjoy yourself and eventually finish the game. You then proceed to play other games such as Portal 2 and Left 4 Dead 2. >
(btw, the story broke when I took out an iphone in the middle of a knight adventure):
> take out iphone and shoot a picture
> You return to the beach. > You return to the beach. > > You return to the beach. >The sun rises:This is a>A new world.A>In this>The sun. The sun
I don't mean that flippantly, I'm serious. Sounds like GPT-2 might be really good at producing an enjoyable experience, even if there's no underlying structure. What would be really cool would be to add some sort of generated structure as well. Can GPT-2 be used to stitch together plot elements, or would a naive Markov model work for that?
https://en.wikipedia.org/wiki/The_Diamond_Age
There are some interesting ideas about what makes a story "engaging", even if there is little or no consistency.
(The book is really good by itself, regardless if you will use it for your project or not...)
And stuff like "There is someone to your left" -> go right -> "you go right towards the person".
Also sometimes the game seems to do something completely different from what I say. Is it actually reacting to my commands, or is it just doing what it thinks the player is likely to do, and then retroactively merging the resulting text with what you initially commanded it to do?
You are , a wizard living in the kingdom of Larion. You have a staff and a spellbook. You finish your long journey and finally arrive at the ruin you've been looking for. You look around and see that it's not much different than when you left it. The only thing is, there's a large hole in the ground where the road used to be
> rename the kingdom to Snabba
"I don't think I like this name," you say aloud. "It sounds like something from a cartoon".
The people seem surprised by your choice. They all ask what you mean. You explain that it's an old word meaning 'stupid
> what is the name of the kingdom?
"Snabba Nul", you reply. Everyone seems to like the new name. You decide to stay here for awhile until you can find another place to live.
What data set did you use to fine-tune your model with? Is there some corpus of adventure game text out there somewhere?
When I try to play it, I get a message at the bottom of the screen saying:
> This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each cell of this notebook manually using Shift+Enter or the run button.
But I don't understand how to do what it is saying.
After logging into Google it seems to be working!
There is absolutely no need for that requirement. If something requires you to log into google its generally a good reason to not log into it.
You dig with your shovel and you find a hole. You fill it with dirt and you stand in the middle of the hole. You stare at the sky and you try to figure out how you got there. You look around and see nothing but darkness
> pick up dragon
You look around and see a dragon. You pick it up and examine it. It seems to be made of metal and has a horn on its head. You wonder if you should use it. You look around and see nothing but darkness
> pick up magical sword
Woops that action caused the model to start looping. Try a different action to prevent that.
> pick up peasant
You look around and see a peasant. You pick him up and examine him. He seems to be made of straw and he is crying. You wonder if you should use him. You look around and see nothing but darkness
You grab the lizard by its neck and pull it close to you. It lets out a scream and tries to escape, but you catch it and hold it tight. You begin to choke it until it falls unconscious. You take off the lizard's skin and sew it onto your own body
taken a step too far there buddy
But to me, this makes very little sense. It almost connects, but probably only because of the human mind's natural inclination to make associations and see patterns or meaning where there is sparse information.
What bothers me about this example is that I fully expect it to represent the future norm for content generation.
It's a bit like modern pop music. We are well beyond the point where software can not only write the music, but even take the once human voice and recreate it to sing the song. Of course it's not great, if one were to really listen - but it is passable, and it makes money. So it drowns out the remaining bits of real human creativity.
Add some Grover-like text for the lyrics (https://grover.allenai.org/), and you're most of the way there. Then use a deepfake singer ( https://youtu.be/k3qJOPrJOQ4 ), and you're most of the way to done.
Depending on the audience, particularly their level of attention or focus on the music, you can probably get away with playing a completely un-human pop song that people will like.
https://slate.com/technology/2019/08/vocal-deepfakes-music-h...
I find the possibilities here very interesting, but I also greatly fear that this will become the norm the same way that nearly every modern song uses autotune.
That's still not enough, and GPT-2 in particular has been heavily overhyped, with fanciful claims that people might use it to generate fake news. (What would the point of that be? You don't need N fake news articles to reach N people, only one, which can be written by a human.)
But it's progress. GPT-2 still feels like science fiction to me. What if a future text generator, maybe even one or two decades down the line, surpasses GPT-2 to the same extent that GPT-2 surpasses the earlier attempts I mentioned? What if that system extends the length of coherence to reliably cover entire articles and essays? What if it gets better at synthesizing the information about the world represented by its training data, so that its output on nonfiction prompts is factually true, rather than mere plausible-sounding nonsense? Is it possible? Perhaps not. But it seems a lot more likely to me now than it did before GPT-2 was created.
[1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/
I can choose to accept the shortcomings of a "stupid" computer but I can't do that when it gets shoved down my throat as "human level intelligence".
Of all the stories in this thread, this moment had me laughing uncontrollably at work. Thanks for sharing
(With appropriate training, of course.)
It runs out of memory while "initializing AI Dungeon", but if I solved that problem would it work? Or is there more that is needed?
Quite amusing that the most GPU-intensive game I want to run on my computer is a text adventure :)
Why can't it be completely sandboxed? Why do I even need to be logged in anyway?
Of course I don't expect that you have any interest in my data, but there's still a granularity problem here regarding access.
I'm cracking up at the Assassin's Creed time warp loop it through me.
Hopefully this can be integrated into full games someday, it's so much more open-ended than any game I've ever seen, despite the enormous plot flaws.
You accept Nic Cage's invitation and become part of his noise band. You soon learn that Nic Cage is a bit of a loose cannon. His shows often end in chaos and he doesn't care one way or another about the people he's hurting. But he sure does love music
The rest is here: https://pastebin.com/yC0FjF96
But I also think the brutal logic really shines a light on how dangerous AI can really be.
Edit. It's the usual local-syntax vs global-content problem. The syntax looks like English, which is an achievement, but the content lacks any larger sense of narrative form, contextual awareness, or subtext - all things you need to make an interesting readable experience.
EDIT: Any chance you can support TPU on Colab?
You open up your laptop and type in your name and hit enter. You get an error message though, saying that you need to install Adobe Flash Player first.
> burn my laptop
You decide not to bother with installing Flash Player and instead just burn your laptop. When you turn it on you get an error message again telling you that you need to install Flash Player. This time you choose to ignore it and proceed with burning your laptop
Sounds pretty accurate!
"I am the Dragon," he says. "My name is Nolazir".
> Hi Nolazir, how are you?
Hi Nolazir! How are you? I'm doing well. What's your name anyway? Nolazir, that's it. Nice to meet you. What do you mean? You're not a wizard or anything
> Have you heard of Larion?
No, but I've heard about dragons. They're pretty big.
> Are you a dragon?
I am not a dragon. I don't even know what they look like.
--------------
hmmm...maybe this is a trap?