> The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.
The AI has trained on the entire factorio wiki along with every tutorial video ever made and who knows what else. I don’t see room for hidden anything in that scenario.
Hah I dunno, I think most human players wouldn't find that very fun and would instead take that as an opportunity to refactor the factory for optimized performance (i.e. the core gameplay loop)
When you just need a few parts to be made, it is sometimes very tempting to drop a chest, fill it up with parts, then go do something else, one of the 1000 tasks you have been neglecting.
That's definitely true, but only to a certain point, after a while it becomes too monotonous to dig around in chests building and combining stuff in an ad hoc manner. LLMs have no problem with that kind of repetitive minutia though.
That was exactly my thought when reading the headline, before even looking into the article. Sure it's still somewhat complex to get to a rocket launch, but without the biters where is the challenge, other than complexity? It's like a big excel sheet with conveyor belts, pipes and electricity poles at that point only.
The more interesting challenge is the one you eliminated, complexity, in getting efficient logistics and arrangements. Dyson Sphere Program (until late in its existence) and Satisfactory don't have waves of enemies attacking your bases but remain challenging and a satisfying puzzle. I remember I used a constraint solver to help me optimize some late game recipes for Satisfactory.
The biters are just a line item in the spreadsheet. Early players might build a wall with an ammo belt, for example. There's lots of options that don't involve responding in real-time.
Though these days I just manually clear the pollution cloud with a pocket full of turrets. The forward deployed engineer model, if you will.
The biggest thing the bugs introduce from an LLMs perspective is a necessity for realtime response. Without them, the LLM can take as long as it wants to decide and do its next operation on the factory. With them, not responding fast enough results in death (particularly early game where a lot of biter combat is player driven). I presume it doesn't do great at this.
The nests stop being a concern once you have significant production and a few hundred logistics bots. Then you just build walls beyond the pollution range and place a gun turret every other square (or lasers at level 10 or above). I only use nauvis for uranium now. Late game you rarely go back.
The issue is most of these play-throughs happen on “normal” response speed. Way too slow for these LLMs to send back and forth and process this many tokens in a quick enough reaction time.
I’ve yet to see running these LLMs on Ultra-fast (8x speed on Sol). Wondering if that’s fast enough to do real time
I hope to experiment with using classic tabular reinforcement learning in game AI one day. Imagine a framework that makes setting up small Q-learning tables easy, and you can just sprinkle dozens of Q-learning algorithms throughout your traditional AI scripting.
These old basic algorithms often have mathematical proofs that they will converge to optimal behavior, and they do so fairly quickly. Turning the complex world of the game into a Q-table would require some creativity and judgement, but is not that hard I think.
In the end we would have a controllable AI that actually does adapt. That's my hope anyway.
With the aid of SOTA LLMs, you can encode specialist intelligence into computer programs without going through the roundabout route of training small LLMs. And even when you do go the route of ML, it doesn't necessarily have to be an LLM or even a Neural network.
with better being more convincing, rather than just stronger
its the same for like a chess bot. i dont want a perfect stockfish that occasionally makes a serious blunder when a dice rolls a certain way. i want one that actually falls into traps
i think hooking in an llm to dwarf fortress for turning the very mechanistic descriptions into prose. same with the adventure mode talk. instead of a constant "well weve got beasts" an llm could read a bunch of the language tree to make a unique introduction for each.
same with the df artworks. get kitfox to design some art styles for dwarves, elves, goblins, and humans, and then instead of just reading
"this is a fine engraving of a dwarf. it concerns the events of 102 when ulrist fastbuttons was elected mayor. ulrist fastbuttons is sobbing"
There has to be a AAAA studio building the new, greatest MMO, that also utilizes LLMs as part of the game loop, surely?
(Not sure how prohibitive it would be to do anything interesting with local LLMs of any kind in a big game, given how many platforms/setups you have to make the stuff run smoothly on, and the sheer amount of RAM they would bind)
https://proxiai.com/ Will Wright (The Sims, etc) is dabbling with AI in his latest game. No idea if it'll be any good, but there's def companies out there trying.
not taking risks is the biggest risk in the times when the most anticipated game drop (gta 6) is probably going to be decompiled within days if not hours of release
LLMs could be interesting geopolitical roleplayers if instructed to faithfully assume the leadership traits or cultural biases of the their intended role, especially in games with expressive diplomacy systems like Stellaris or Europa Universalis. It could also make the diplomacy experience a lot richer because the LLM is able to interpret ligusitic tone and other verbal cues. The modern diplomacy experience is basically just a dialog tree with stats and dice rolls attached to it.
Unfortunately for it to be sustainable it would need to be an LLM with VRAM usage probably under a couple gigs max so it can run locally on the gaming PC. That said it may well be better than current AI in strategy games.
Yes, the insane hardware requirements make it totally impractical - that's why I say "turn based" because turn based games would at least allow for a practical gameplay experience even with remote LLM APIs.
impossible, unfortunately, because LLMs suck at creative writing. without handholding, they are incapable of producing anything but cliche-ridden purple prose for young adults. you can feed it the first half of a book with lots of sex, violence, and profanity, ask it to continue the story, and it will degrade into safe slop within two chapters.
both American and Chinese labs aim their models at coding and knowledge work, so they penalize adult themes in post-training.
Fable seems to be a potential franchise to do all this; imagine a "living fantasy world" where everything truly unfolds based on your decisions, giving NPCs all the functions/actions that a player has.
Not sure about LLMs but I yearn for someone to train neural nets to play the NPCs in a shooter game well but without having aimbots or seeing through walls - basically a “true” version of F.E.A.R. AI where they play simulated squads that overwhelm the player with various tactics and adapt to their playstyle. Or even just make increasingly complex state machines and goal oriented systems which might be more performant but still encode the main rules.
It is weird and almost alien how AI designs factories. When Claude announced Fable 5 they also did a Factorio demo that was the same kind of designs and a lot of hand feeding and direct insertion.
In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution. But I honestly expected more from frontier models
I think one of the problems is that AI has no sense of laziness or time. Unless it's in their training data or prompted they're not going to bother automating something
Actually if you estimate the time required it's plausible that player feeding comes out on top, especially if setting up automation is not that obvious. But human player would never play that way, they would get bored from the repetitiveness, and the micro scheduling would be a headache. So the game designers never really have to close that gameplay loophole (or closing it would make human experience worse).
I have a similar problem on my GK2 vibegamed playthrough (see https://news.ycombinator.com/item?id=50302771), if I want computer to play similar to how a human would, I need to artificially close that loophole (for example, AI prefers to gather/eat low-tier food to cooking better).
> In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution.
Is the weirdness of the designs meaningfully beneficial in any way that the AI has figured out, or could it for example be that the training data the LLMs learned from are too different from the type of data it would need to learn from in order to interface properly and play the game better? And then because of that incompleteness, the LLM has to hallucinate a lot and it results in weird outcomes.
To them the player character is just as good as a belt for transporting materials.
The player character actually does have an extremely high capacity so if you are constantly running between chests your factory will still be reasonably high throughput.
It doesn't seem to understand the core mechanic of Factorio, which is putting outputs onto belts and routing them to other inputs. It does this in a few places, but for the most part everything it put into chests and moved around by the player.
It's just easier to do manual stuff when the complexity of scaling the automation becomes cognitively demanding. A new player does mostly everything manually while experienced players have a natural intuition for planning and scaling the automation.
If you don't have enemies, you have unlimited time. Doing it thru player is what it learned and wasn't pressured to do better (it has no fatigue as opposed to humans).
If you measure by the product of throughput and distance traveled, for an item with a stack size of 100 (e.g. metal plates), the player's inventory adds up to roughly the same capacity as 2500 yellow belts. And that's assuming 100% utilization running back and forth in both directions, with no unused inventory space and no wasted time.
A basic assembler can manufacture 2500 belts in about 21 minutes. So as a rule of thumb, if you're playing for longer than 21 minutes, belts are worth the investment (at least for bulk transport of raw materials).
(I guess maybe add an additional factor of at most √2, since the player can run diagonally and belts are orthogonal.)
This is a much better benchmark as a concept (without knowing the details of how the problem is framed to the agent) than some of these ridiculous things the frontier labs are reporting (vending machine simulator? come on).
Can you report more in depth at stumbling blocks? It's a surprisingly linear curve.
I'm going to try it on a local run.
I wonder if there is room for "recommendorio" where it gives you AI advice on your current factory.
Is the model pre-trained with any knowledge of the game?
If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly.
I suspect Opus 5.5 can do even better but there aren't as many people testing it.
How do they feed the game screen to the models? I imagine the models are too slow if you send 60FPS, but I feel like that’s also the case if you maybe feed it 1FPS? If you go any slower, you’ll miss too much info?
One limitation of these type of demos (and similar with Minecraft, etc) is that:
1. Putting into "peaceful" mode makes response time a non-factor
2. These models have read and have access to every guide and video ever published on the subject. Most humans, especially on first play, learn game systems and interactions from scratch, then develop novel strategies and plans and "hypothesis test" their way into beating the game.
This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great.
At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).
I don't have the budget to do so, but if I do, it would be interesting to put different models in the same game with the same prompt to observe the behaviors that different models have when told to problem solve.
82 comments
[ 7.0 ms ] story [ 43.2 ms ] threadRobots: they're just like us!
youre not really sure how to set up some of the logistics/how much of what you will need where ahead of time
Hah I dunno, I think most human players wouldn't find that very fun and would instead take that as an opportunity to refactor the factory for optimized performance (i.e. the core gameplay loop)
When you just need a few parts to be made, it is sometimes very tempting to drop a chest, fill it up with parts, then go do something else, one of the 1000 tasks you have been neglecting.
Though these days I just manually clear the pollution cloud with a pocket full of turrets. The forward deployed engineer model, if you will.
I’ve yet to see running these LLMs on Ultra-fast (8x speed on Sol). Wondering if that’s fast enough to do real time
These old basic algorithms often have mathematical proofs that they will converge to optimal behavior, and they do so fairly quickly. Turning the complex world of the game into a Q-table would require some creativity and judgement, but is not that hard I think.
In the end we would have a controllable AI that actually does adapt. That's my hope anyway.
its the same for like a chess bot. i dont want a perfect stockfish that occasionally makes a serious blunder when a dice rolls a certain way. i want one that actually falls into traps
same with the df artworks. get kitfox to design some art styles for dwarves, elves, goblins, and humans, and then instead of just reading
"this is a fine engraving of a dwarf. it concerns the events of 102 when ulrist fastbuttons was elected mayor. ulrist fastbuttons is sobbing"
(Not sure how prohibitive it would be to do anything interesting with local LLMs of any kind in a big game, given how many platforms/setups you have to make the stuff run smoothly on, and the sheer amount of RAM they would bind)
both American and Chinese labs aim their models at coding and knowledge work, so they penalize adult themes in post-training.
Boring though without something to attack you imo
I have a similar problem on my GK2 vibegamed playthrough (see https://news.ycombinator.com/item?id=50302771), if I want computer to play similar to how a human would, I need to artificially close that loophole (for example, AI prefers to gather/eat low-tier food to cooking better).
Is the weirdness of the designs meaningfully beneficial in any way that the AI has figured out, or could it for example be that the training data the LLMs learned from are too different from the type of data it would need to learn from in order to interface properly and play the game better? And then because of that incompleteness, the LLM has to hallucinate a lot and it results in weird outcomes.
The player character actually does have an extremely high capacity so if you are constantly running between chests your factory will still be reasonably high throughput.
Part of the reason players try to automate manual feeding is the annoyance factor, which AI doesn't care about.
Stack size varies, but let's take 75 as an average, which means a player can carry 5600 items.
But carrying isn't throughput. If a round-trip takes 30s, player throughput will be 186 items/s. That's a measly 12 T1 belts.
If you measure by the product of throughput and distance traveled, for an item with a stack size of 100 (e.g. metal plates), the player's inventory adds up to roughly the same capacity as 2500 yellow belts. And that's assuming 100% utilization running back and forth in both directions, with no unused inventory space and no wasted time.
A basic assembler can manufacture 2500 belts in about 21 minutes. So as a rule of thumb, if you're playing for longer than 21 minutes, belts are worth the investment (at least for bulk transport of raw materials).
(I guess maybe add an additional factor of at most √2, since the player can run diagonally and belts are orthogonal.)
Can you report more in depth at stumbling blocks? It's a surprisingly linear curve.
I'm going to try it on a local run.
I wonder if there is room for "recommendorio" where it gives you AI advice on your current factory.
Is the model pre-trained with any knowledge of the game?
I suspect Opus 5.5 can do even better but there aren't as many people testing it.
This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great.
At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).