HN Explorer

Show HN: AI-town, run your own custom AI world SIM with JavaScript (github.com)

429 points by ykhli ↗ HN
Hi HN community! We want to share AI-town, a deployable starter kit for building and customizing your own version of AI simulation - a virtual town where AI characters live, chat and socialize.

Inspired by great work from the Stanford Generative Agent paper (https://arxiv.org/abs/2304.03442).

A few features: - Includes a convex.dev backed server-side game engine that handles global state - Multiplayer ready. Deployment ready - 100% Typescript - Easily customizable. You can fork it, change character memories, add new sprites/tiles and you have a custom AI simulation

The goal is to democratize building your own simulation environment with AI agents. Would love to see the community build more complex interactions on top of this. Let us know what you think!

Demo: https://www.convex.dev/ai-town

I made a world Cat Town to demonstrate how to customize AI town. Using C(h)atGPT :)

Demo: https://cat-town.fly.dev/ Code: https://github.com/ykhli/cat-town

120 comments

[ 3.5 ms ] story [ 191 ms ] thread
huevosabio
This is fantastic, will try it out this week! Thanks!
jmorgan
If you haven't yet checked out the Generative Agents project referenced by OP, definitely give it a look, it's open source: https://github.com/joonspk-research/generative_agents

Over the weekend Lance Martin got it working with local models using llama.cpp and ollama.ai which saves $ on longer sims since all inference happens locally https://twitter.com/RLanceMartin/status/1690829179615657985. It's neat how the AI agents interface with each other – e.g. one will host a party and invites will be sent throughout the group

calebio
would be really fun to hook this up to Stardew Valley!
andai
I wonder how llama affects the simulation. Even llama2 consistently fails simple reasoning tasks.

(Though I suppose people aren't very reasonable, so it might give an accurate simulation!)

afriday11
Congrats @ykhli! Excited to check it out.
jedberg
TIL a16z has a GitHub repo with a bunch of cool stuff.
dadrian
If an a16z repo gets enough stars, will they invest in themselves?
adolph
Fund of funds making funds where market research generates revenue and everyone’s adventures are capitalized into an on chain Large Lifestyle Model where being Day One is to emit content enough past the zeitgeist to avoid banality but not so far as to be incomprehensible.
gabereiser
cells within cells within cells within cells...
quickthrower2
Not only that this looks like a toy.
anonzzzies
And not a great one ; the sprites walk into the sea.
ShamelessC
How else would they have “skin in the game”?
[deleted]
(comment deleted)
xwdv
What does it mean for them to “live”? Are there certain goals they try to accomplish? Find food? A mate? Build families?
jameshart
We've been studying humans for thousands of years and are no closer to answering those questions for them. Why do you think we'd have an answer for these newfangled AI agents?
lacrimacida
Responding to a question with another question doesn’t help the op who seemed to be genuinely asking an honest question about this AI simulation that this while thread is referring to. And second, I think we do know a whole lot more about ourselves than we do about AI if your question wasn’t supposed to be rhetorical
SamBam
I would assume that, as the developer of an agent, you could create those goals for them.

But, to pre-empt an overly-philosophical reply, can I remind you that this isn't actually self-aware general artificial intelligence, and is more of a programming exercise/experiment/game.

testernews
This is great! How do i self host it (no open ai)?
tudorw
I think this is the rabbit hole you're looking for, https://twitter.com/RLanceMartin/status/1690829179615657985

note you still need pinecone (vector db) and something called clerk.com that does auth, not sure why, seen other projects swap out pinecone for chroma, and not sure why we are authorising things with clerk.com, but that cannot be essential...

tpae
What happens after you trap them for certain generations, and all of the sudden give them access to the real world? Would their "minds" break?
pkiv
The AI-town stuff is cool, but the real benefit is how all the backing frameworks are already integrated. I'll definitely be using this as a jumping-off point for my next LLM project.
felideon
How long until the agents recreate AI Town in-universe? We wouldn’t be able to see it, but one of them could start talking about it. ;)
drewtato
I looked at the demo and apparently everyone is a sociopath? One character is literally described as "a sociopath who has no empathy". So it might be a good idea to check up on them every once in a while.
haolez
Maybe I'm weird, but I can imagine this being 10x more fun with a completely uncensored and toxic LLM backend :)
[deleted]
(comment deleted)
liorben-david
This is awesome!

Game idea to build on top of this: Table top deception type game where each agent has the goal of convincing the real users that they are in fact also real users.(So each agent is trying to pass a turing test).

Every AI agent uses RL to optimally prompt their personal LLM for how they should chat with the human players. eg should they try to frame a certain person, should they play it dumb, should they gaslight etc.

huevosabio
I think it may be even more fun the other way, players need to rat out other players, do they have to pretend to be an AI.

It makes it easier for the AI mods to do the part and it puts the burden on the players.

reaperman
Yep. "Find the AI" is broken for now, because whenever I've played it, humans can just be exceptionally rude/lewd or use super-modern slang. Making the humans try to blend in as AI is much more interesting, as a game.
eloisius
A game like Press The Button, except there’s half AIs and half human players. The goal for each is to identify the others and airlock them off the ship. Constrain the tests in such a way that open chat using lewd language or whatever is impossible. I’d play that once or twice.
Kiro
Where do you play that?
wishfish
I like your idea of find the human. Just building on that idea a little. I know current AI detection programs don't work well. But they would be fun in the context of a game. Call it "Only Robots Allowed" and have it be a single player version of Among Us. Pretend to be a robot while trying to sabotage robot things. AI detection is applied to your conversation with other robots. And also applied to your movements. If you fail the AI detection by not emulating an AI well enough, then it's time for "kill all humans!"
falcor84
This is probably the most important life skill we should be teaching in schools </pessimism>.
pelorat
It would be pretty easy to beat. One sure way to tell an LLM apart from a human is to type something nonsensical and optionally repeat it multiple times. A human would inevitably answer with something along the lines of "what the hell are you on about dude???", which is something you'd never see from an LLM.
e12e
> which is something you'd never see from an LLM.

Might not see it from ChatGPT - but "never from an LLM"? Why would you think that?

gmerc
So you make it learn. It collects everything humans say and matches it to the situation, replaying human conversation as needed.
dalore
So like Battlestar Gallactica. Who is the cylon? The cylon might not even know they are not human until revealed to them later.
chaoz_
With the current state, most powerful LLMs are also limited in the amount of topics they are allowed to discuss. I bet you could easily differentiate between player and AI by asking opinions on some controversial topic.
pablo24602
I built something very similar when the paper came out- watch out for the instances where GPT returns both sides of the conversation (e.g. interpreting a prompt with chat history as being representative of what the output should look like, and putting out both "Jack" and "Jill"'s text). The first sample I went to suffered that problem (output from "Lucky's perspective below.)

"Hey Stella! It's so great to see you! You'll never believe what I've been up to recently. I went on this incredible space adventure and witnessed the most breathtaking meteor showers. And guess what? I even encountered alien beings who communicate through colors! It was absolutely mesmerizing. Stella: Wow, Lucky! That sounds absolutely fascinating. I've always been intrigued by space and the wonders it holds. I would love to hear more about your adventure. Lucky: I'm thrilled that you're interested, Stella! I can't wait to share all the details with you. But before we dive into that, I heard you're passionate about gardening. Is that right? Stella: Yes, that's true! Gardening is one of my favorite hobbies. I find it so peaceful and fulfilling. Lucky: That's fantastic! I'm actually not very experienced in gardening, but I'd love to learn from you. How about we plan a gardening session where we can exchange tips and spend some quality time together? Stella: That sounds wonderful, Lucky! I'd be more than happy to share my gardening knowledge with you. Let's find a time that works for both of us. Lucky: Absolutely! I'm really looking forward to it. But hey, no rush. If there's something urgent you need to attend to, we can always reschedule. Just promise me we'll find another time to connect. Stella: Don't worry, Lucky"

drusepth
Did you have any luck mitigating this problem?
baobabKoodaa
It's relatively simple to detect this type of defect and handle it during/after generation.
SamPatt
I've run into this issue a lot with ChatGPT, and almost never with GPT-4. I know it isn't always possible, but just using GPT-4 prevents this 99% of the time (basically 100% with proper prompting).
fennecfoxy
Sure, to force chat gpt to role play I've found that giving it a character to play and then prefixing your query with underscores for meta-roleplay stuff works well.

Ie if the LLM is playing a character "Jack" and you are playing the character "James" your query might be "_only reply for your character Jack_ James: I pick up the sword and then turn towards Jack".

It can also be used to influence behaviour as LLMs often get stuck repeating (not word for word) the same events/description, ie greeting your character over and over again and not moving on, LLMs aren't great at having autonomy/agency in the flow of a conversation, I think this is best done by not providing the entire history of the conversation but instead distilling it to relevant information for the current query.

But that can also be mitigated manually by bumping their character with underscores ie "_Jack asks what James wants to order_ James: I return Jack's greeting and peruse the Tavern's menu board".

dalore
You make the other side of the conversation the stopword. i.e. "Jill:" for Jack and Jill.
swyx
what a descent from loving nature down to "heyyy scheduling is hard"
dalore
I did that but the trick to fix that was to use the other person's name and : prefix as the stop word. So in your case "Stella:". Then if it tries to continue the other side conversation it would just stop.

I also noticed this when browsing the demo village, some conversations it ran through and did both parts.

pablo24602
This ended up being my solution as well- although I still ran into edge cases where GPT would output things like "Stella said". Never did manage to 100% fix this issue, even with creative prompts.
andsoitis
> a virtual town where AI characters live, chat and socialize.

Is it meant to be like a zoo, where humans gawk at other creatures?

firtoz
You can interact with them, in the original paper, at least.
andsoitis
I doubt they’ll be interesting.
moomoo11
Could lead to cool sim games.

I’d love a modern zoo tycoon. I enjoyed that game as a kid. Or roller coaster tycoon.

There you go, hopefully y’all make some money. Just send me the games you help to make on steam lol.

drusepth
A friend and I recently started a game studio that was largely inspired by this paper. This is an amazing foundation and I'm excited to build some more complex strategy games on top of it.

If anyone else is doing similar work applying these concepts to consumer gaming, I'd love to hear about what you're doing (dru[at]chromagolem.com)!

leke
This is great. I love how weird some of the characters can be, just like in real life. For example:

Pete is deeply religious and sees the hand of god or of the work of the devil everywhere. He can't have a conversation without bringing up his deep faith. Or warning others about the perils of hell.

Kurt has something to hide. It obsesses him and colors everything he says. He's so afraid someone will figure out that he is obviously evasive. He'll never tell anyone the secret, but he'll ellude to it alot. It tortures him. And his life has become a mess as a result of it.

Stella can never be trusted. she tries to trick people all the time. normally into giving her money, or doing things that will make her money. she's incredibly charming and not afraid to use her charm. she's a sociopath who has no empathy. but hides it well.

---

To take this to the next level, I hope you would be able to prompt your own characters, and perhaps have places you can send these guys on holiday to converse with other people's characters.

Also, I think this would be great as a tool for learning foreign languages. Just because it's interesting, engaging and based on language. Again with prompts that can be programmed, like Gill who constantly talks about his job in marketing, and Bill who likes to refer to himself in the third person, and Betty who constantly uses conditionals in her sentences. Again, just so cool.

jamesdwilson
To be crass and direct,

> The goal is to democratize building your own simulation environment with AI agents. Would love to see the community build more complex interactions on top of this. Let us know what you think!

why? what's the point?

tomcam
The mission statement you quoted is incredibly clear. Very often both commercial and open source projects announced here on hacker news have completely impenetrable descriptions or mission statements. That is not the case here. I literally could not improve on it without quoting it directly.

What is unclear about the portion you quoted?

jamesdwilson
why did they make this? what is the purpose? why do they want "to see the community build more complex interactions on top of this" - what's in it for them?
[deleted]
(comment deleted)
srackey
ITS FUN!!!

And maybe can be used to do interesting things.

Let’s work on the cool tech together.

I promise you it will not hack into the mainframe and turn the world into paperclips. (Odds are…near zero).

parallel
I'm making a conscious effort to do more things that have no extrinsic objective, to do things just for their own sake. This is a familiar concept, it's a hobby (or art?). I feel I'm missing some of the joy and satisfaction that life has to give by always needing a stated tangible objective.
jamesdwilson
This is made by a16z, ostensibly for business reasons. I am asking what their angle is.

As far as I know, they don't have much history in making things with "no extrinsic objective" - I could be wrong though.

Kiro
You misunderstand what a16z-infra is. This is built by Convex, the game engine being used.
fnordpiglet
Cooperative agents have a lot of utility in more applied settings and the interesting part here is the agents are autonomous agents operating over an abstract semantic space using LLMs unique ability to “reason” abductively. The key to my mind in these exercises is the construction of constraints, management of context, and delegation to task based optimizers (i.e., the agent has a goal, the LLM abstractly constructs a solution given the context, and optimizers like path finders carry it out)

I’ve thought a lot that these sorts of techniques can be used to make things much more generalized. A multimodal model backing say manufacturing equipment can make the equipment much easier to reconfigure to new tasks in an environment that’s more adhoc and includes other machine and human agents.

The key to my mind is the fact it’s not just LLMs it’s a mixture of techniques - constraints, optimizers, information retrieval, etc.

More specifically the demo involves a lot of Andreessen Horowitz investments to demonstrate them in an accessible way that people find interesting because it’s a game. And people like games.

jamesdwilson
Thanks! Can you please elaborate on "Cooperative agents have a lot of utility in more applied settings?"
fnordpiglet
Imagine being in your shop and telling a robot to get you a hammer. It can understand the instructions, look at the environment using an inverse image model, synthesize the environment with the instruction, delegate to path finders etc a route to the tool box and retrieve you a hammer without ever being programmed anything. Now abstract you to being other machine agents in the environment of a workshop being instructed goals along some plan. Today factory robots are highly specialized machines that require a very finite and controlled problem space and reconfiguration is expensive and time consuming, if possible at all.
jamesdwilson
Thank you very much @fnordpiglet for addressing my questions directly and thoroughly! Makes a lot of sense. I still don't understand why a16z wants us to do that though. what's in it for them?
fnordpiglet
They’re showcasing a bunch of portfolio companies in their integrations. That’s why the setup is 300 steps long.
Imnimo
Here's a chat message from a random character in the demo I clicked on:

ALEX

8/15/2023, 1:53:43 AM

Absolutely! Here's a glimpse of my latest masterpiece. [Attaches a photo of the painting] What do you think?

===

I feel like it will be difficult in general to prompt the LLM in a way that gets it to stick to the limits of the simulation environment.

tavavex
This is a very interesting use case of the Generative Agents project.

I feel like this is just one of the earlier attempts at creating an interactive experience with this, but there's still so much potential for future games, if the technology is refined further. Think of something like Animal Crossing (or any other game where interacting with NPCs is at the forefront of gameplay), but with the characters that have completely unique personalities and an ability to generate new dialogue (rather than relying on a limited number of pre-written lines), or react to player actions in unique ways, and so on.

fnordpiglet
I feel like this might be the beginning of a much richer single player game genre that mixes generative language, image, texture, and other features with procedural rogue like world generation with the semantic features defined by the LLM. Obviously NPCs, dialogues, etc, can all be context aware agents bootstrapped by the environment and player history. Content creators IMO will be creating assets that are used to fine tune and create LoRas as future game assets. Instead of monotonously creating every asset in a game, they would create styles and control nets, language corpuses, and human written lore RAG’ing into game context, etc. I think artists will have a wonderful contribution much richer than simply painting a picture.
pyinstallwoes
Which ultimately ends with thinking as a generative prompt into thoughts which get passed to the universal rendering engine known as reality.