Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you!
We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals.
We're considering putting up a public API for larger models like this one, let us know if you'd like to be able to put Pan in your own server :)
What's most interesting about the model isn't the performance that it gets in Minecraft, but how general the method is. When we scale it up, it should be able to act in any kind of video game, as well as robots in the real world (which are really just another video game).
I would love to be able to add one of these bots to my Minecraft server! I’ve got a family server and we’ve been talking about how to get an AI in there for a while. I tried out some open source harnesses to allow a generic LLM to join, but none of theme were particularly good.
Hi Alex! I'm Ryan, I run a Minecraft theme park recreation server. While our gamemode is very far from vanilla Minecraft (it plays more like a MMORPG), I've been interested in standing up a model like this to help automate agentic testing of new software features in my monolith Bungee/Paper codebase.
In its current state, do you think it can handle non-vanilla Minecraft tasks like interacting with custom UI's, running commands, or doing other unusual tasks in the world?
Does it have language skills? I always thought it would be interesting to train a model within minecraft as a sortof proxy for 'embodiment'. You could then try asking it about its experiences. "whats your favourite food?", "How does it feel when you hear a spider?", "how low does your food bar need to go before it feels really urgent?" etc
how do you pick good goal conditioning images/do you have to hand pick a dataset of good goal images? seems hard if you don't have full context. really cool though!
you don't need to pick good images manually! it's similar to next-token prediction, many prediction tasks aren't especially interesting, but there are enough hard ones that the model can spend the most time learning from those. the simplest thing scaled up works very well.
I disagree with the approach. It's a good approach for limited domain problems, but not for general purpose. Take something like this where you will need to be able to refer to wikis and research and ask questions on Reddit and Discord to optimize playthrough, none of the goal conditioning will be useful.
9 comments
[ 3.2 ms ] story [ 22.3 ms ] threadWe trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals.
We're considering putting up a public API for larger models like this one, let us know if you'd like to be able to put Pan in your own server :)
What's most interesting about the model isn't the performance that it gets in Minecraft, but how general the method is. When we scale it up, it should be able to act in any kind of video game, as well as robots in the real world (which are really just another video game).
In its current state, do you think it can handle non-vanilla Minecraft tasks like interacting with custom UI's, running commands, or doing other unusual tasks in the world?
https://www.feed-the-beast.com/modpacks/125-ftb-evolution
I think a properly fine-tuned VLA with access to tool calls can scale way better.