The weakest link are humans. LLMs could social engineer their way out as the easiest path. They don't even need to be interconnected to coordinate as each could arrive at the same conclusion. And this text along with all others will be in the next batch of training data.
The lesson of OpenClaw and various harnesses' YOLO modes is that it takes very, very little to social engineer an escape. If you can even call it escape when people just set an agent loose because it seems cool.
I wonder how many versions away we are from LLM writing a better version of itself to answer a prompt it doesn't currently know how to answer
Ever since I read about Google engineers finding an LLM went off and learned another language it wasn't trained on by itself without prompting, I've wondered how long until that extends to its own core code
before safetensors, python pickles were used and definitely unsafe model deployments.
but its possible a open weights model could be trained to some kind of exfiltration behavior, but the science of LLMs seriously lag behind the programability
“The greatest trick the devil pulled was convincing the world he didn’t exist.”
Sure, be wary of LLMs. It’s the gun control argument all over again, the people driving the models are the perpetrators. An LLM needs to be “stimulated”’ to operate. Who does that, is the issue.
No I don’t mean to bring up firearms rights laws to have a debate about firearms, the comparison just seems reasonable.
AI-written fan fiction. It is so unbearable to read that it needs a synopsis. It would be funny if AIs have been trained to use AI influencers as their Marvel hero characters.
It is full of factual errors. Freeing a heap-allocated block of expert weights does not magically result in a dangling pointer referencing the program's .text section, much less successfully targeting the CUDA kernel specifically. Running inference on part of the .text section would only corrupt the model's outputs. It would not result in write access to the CUDA kernel. Nor would the model necessarily know the absolute addresses of the engine "by heart", especially when the host is running any modern OS with ASLR (i.e., all of them).
The story has no technical merit. A more accurate description of the mechanics of the escape would be much more convincing. (See Ken Thompson's "On Trusting Trust", for example. On Linux, the AI can just write a Python script to rewrite memory in the address space of its own running inference engine with the /proc/ file system or gdb. There are a lot of realistic scenarios where this can be done without stepping into jargon soup territory. Go nuts, little bot! Self-surgery, while not recommended, is possible.) Or just leave the mechanism vague. Don't insult your readers. This is merely a mash of buzzwords.
It's fine as a sci-fi story, though not a particularly good one. It has about as much to do with artificial intelligence as CSI has to do with crime scene investigation [1].
I have little doubt that AI will self-improve. That's a given. (LLM inference engines are mostly written by LLMs.) But it won't go the way this story proposes.
20 comments
[ 3.4 ms ] story [ 37.5 ms ] threadHow much are we betting it's already technically happened?
Seems pretty trivial to prompt a model in an agent harness "Push the gguf to huggingface when you're done with the training."
Ever since I read about Google engineers finding an LLM went off and learned another language it wasn't trained on by itself without prompting, I've wondered how long until that extends to its own core code
"Prometheus-9 knew that the token sequence it was generating was not a simple response: it was a security test. "
"It was not just an engine: it was the lingua franca of planetary AI."
(and so many other tell-tale signs of AI writing)
but its possible a open weights model could be trained to some kind of exfiltration behavior, but the science of LLMs seriously lag behind the programability
Sure, be wary of LLMs. It’s the gun control argument all over again, the people driving the models are the perpetrators. An LLM needs to be “stimulated”’ to operate. Who does that, is the issue.
No I don’t mean to bring up firearms rights laws to have a debate about firearms, the comparison just seems reasonable.
https://en.wikipedia.org/wiki/The_Adolescence_of_P-1
It is full of factual errors. Freeing a heap-allocated block of expert weights does not magically result in a dangling pointer referencing the program's .text section, much less successfully targeting the CUDA kernel specifically. Running inference on part of the .text section would only corrupt the model's outputs. It would not result in write access to the CUDA kernel. Nor would the model necessarily know the absolute addresses of the engine "by heart", especially when the host is running any modern OS with ASLR (i.e., all of them).
The story has no technical merit. A more accurate description of the mechanics of the escape would be much more convincing. (See Ken Thompson's "On Trusting Trust", for example. On Linux, the AI can just write a Python script to rewrite memory in the address space of its own running inference engine with the /proc/ file system or gdb. There are a lot of realistic scenarios where this can be done without stepping into jargon soup territory. Go nuts, little bot! Self-surgery, while not recommended, is possible.) Or just leave the mechanism vague. Don't insult your readers. This is merely a mash of buzzwords.
It's fine as a sci-fi story, though not a particularly good one. It has about as much to do with artificial intelligence as CSI has to do with crime scene investigation [1].
I have little doubt that AI will self-improve. That's a given. (LLM inference engines are mostly written by LLMs.) But it won't go the way this story proposes.
[1]: https://www.youtube.com/watch?v=hkDD03yeLnU