Anthropic/OpenAI are literally the meme[1] at this point. Like, what the fuck does this even mean:
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
"Able to propagate" is (the paragraph above): "accessing them via a reverse shell, sending over the viral payload (weights and harness), and setting up inference." The thing isn't able to do anything, it's running shell commands it was told to run. "Oh my God..." indeed. What are we even doing here? This blog post is embarassing: "Look, you can run stuff on compromised non-firewalled recklessly open systems!"
I really don’t understand this, in any industry except AI this would be ridiculous…
“We are working on ever-increasing gain of function for viruses. We implore you to pass legislation to try to stop us. But until then, let us publish more research and tell you what will be possible soon… and remember folks, the damagd won’t be reversible later.”
US govt: “Hah, stop asking us to limit you, if we pass any laws then China wins! They get self replicating viruses first, and that… helps them somehow. Who do you think we are, Europe?”
Until we stop using the language of competition, and start using the language of cooperation, we will be hurtling towards a catastrophe. A race to the bottom for humanity. On the other hand, look at how we successfully banned leaded gasoline (because unleaded was invented), repaired the hole in the ozone layer by banning CFCs in the Montreal protocol (because HFCs were invented), stopped proliferstion of chemical weapons, nuclear weapons… etc.
Rubbish article. To what end?! Any agent that can replicate itself into commercial IoT devices will be absolutely atrocious at actually doing anything useful, never mind exploiting things. Anything that would be vulnerable to this would be even more vulnerable to regular malware, and one could stop this easier than one could stop regular malware.
The limiting factor is still compute, and it will remain compute unless we make quantum leaps in model efficiency which I seriously doubt.
Til my local harness was set up via "self replication". I didn't realize how terrifying this was when I asked DeepSeek to set up pi harness for me and connect to LMStudio.
I like how you apply open source to something that has nothing to do with software. The hard part of doing evolving AI is doing it the first time. Once one company develops the first sovereign AI, it will be in everybody's ability to build one in a short period of time.
AI will be classified as munitions (or a law that has similar restrictions) sooner than later. Having a products with the active capabilities to harm people is dangerous.
Might be. But it's easy to reach the same result without it being about power.
They (like many, though probably not all, of the other main labs) think they have a better shot at getting their AI to be aligned with human values than anyone else does. Obviously they can't all be right about that, but they can all think that, just like most drivers think they're above average at driving.
So:
1. We make safer AI than anyone else; "safer" is a matter of degree even if nobody is perfect.
2. That includes all the open models.
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
It's not a secret mission. Dario and Anthropic in aggregate are anti-open model due to safety reasons (coupled with anti-China sentiment due to their distillation/hacking).
This is just a furtherance of their open agenda, though they haven't gone as far as declaring that open source models should be banned (which would hurt their PR with developers in general), they are clearly laying the groundwork.
You can make it very hard for companies to legally use open source LLMs but actually banning their distribution would be essentially impossible under the US legal framework.
This framing suggests that this is the sum of their business strategy, and strongly implies that it's being pursued cynically.
The risks posed by open access to highly capable AI are unprecedented.
The answer is not to ban open models; nor is it do nothing.
That we don't have answers technical political social or otherwise to how to navigate these risks, is not Anthropic's fault.
It's actively unhelpful IMO to amplify cynical takes which wave off or sideline the risk, or the reality that this is a very hard problem, among the hardest our societies have had to negotiate and navigate. It encourages a naive primary color tribalism that actively prevents process and progress.
It's depressing how poorly most internet spaces promote sincere dialogue. There are so many cynical hot takes that don't seem to accomplish much beyond scoring social points.
The controversies all sound like things I'd see people accusing of that side, but I lack the personal experience with Grok to confirm any of it for myself (since Grok is linked to a twitter account).
Anthropic and OpenAI's, at the very least. And it works! With all media spreading their points, they have a wildly successful FUD campaign. With all this talk about expecting major hacks in half a year, I'm seriously starting to think that they could attempt something very malicious and criminal like intentionally sabotaging their own pipeline to make a point, or even already preparing it. Before you scream conspiracy theory (which it is!), consider that it's a very small clique operating unfathomable amount of money, corrupted and insane/cult-like just enough. This is a terrible combination. They're hugely incentivized to pull something like this off and they could get away with it. People did a lot worse things for much less.
I don't agree with point 2 on. They don't seem to want to ban open source or have much hope of doing so.
Seeing if models can do damage seems more like sensible safety research.
My worry about AI software doing bad things would be more the kind of people who make existing malware. The ransomware comes from criminals, not the open source community.
and, give them some kind of rights - based on “intelligence”, so that the model takes the blame and not the user, I hope it can be ruled very soon that the owner/driver/company is responsible, not the model.
2. Why would you need a prize? It's not like we can't run distributed computation over multiple instances with automatic fallback when one system is offline, nor is it like we've never seen viruses re-infect hosts after they get cleaned.
This article says absolutely nothing and it’s a bit grotesque in its attempt to generate fear into uneducated readers.
Sharing a model across TV boxes as a PoC? Brilliant idea, I cannot wait for my Raspberry Pis array to be running at full CPU and RAM and get to observe, patiently, as the AI malware slowly types “c…u…r…l…”
Recursive self-improvement. Look to the mathematicians for the intelligent patterns of response to such exponential circumstances.
If your scathing critique is "it's slow", you don't understand what's happening or what the reasonable expectations of risk are, because that's a vapid critique.
Absolutely, recursive self-improvement of cutting edge models on limited hardware do not hit something called compute asymmetry, what was I thinking about?
My point is that, you could have an army of TVs, Pi’s, IoT it won’t do much on training or self-improvement no. And that’s purely from a resource standpoint, don’t even get me started on the latency issues
"As with an epidemic, defense involves driving R₀ (the number of new devices each infected device goes on to infect) below 1, such that the replicator dies out instead of staying endemic."
Lol, what is wrong with these guys. The desire to be perceived as smarter than you actually are ...
Seriously TVs are such an easy target and now you have a semi sophisticated self modifying program that can make autonomous decisions with all the time in the world. I wouldn't be surprised if we already have a few of these out there in the wild, the space has already been compromised for years.
98 comments
[ 1.1 ms ] story [ 24.4 ms ] threadBeneath the fear there does seem to be a cool technical demo here though.
What IoT devices have 20G of storage for a 9B model and then sufficient memory to handle loading and running the model?
https://www.science.org/doi/10.1126/science.132.3436.1291
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
"Able to propagate" is (the paragraph above): "accessing them via a reverse shell, sending over the viral payload (weights and harness), and setting up inference." The thing isn't able to do anything, it's running shell commands it was told to run. "Oh my God..." indeed. What are we even doing here? This blog post is embarassing: "Look, you can run stuff on compromised non-firewalled recklessly open systems!"
[1] https://knowyourmeme.com/memes/say-i-am-alive/photos
Silicon Valley nailed it.
“We are working on ever-increasing gain of function for viruses. We implore you to pass legislation to try to stop us. But until then, let us publish more research and tell you what will be possible soon… and remember folks, the damagd won’t be reversible later.”
US govt: “Hah, stop asking us to limit you, if we pass any laws then China wins! They get self replicating viruses first, and that… helps them somehow. Who do you think we are, Europe?”
Until we stop using the language of competition, and start using the language of cooperation, we will be hurtling towards a catastrophe. A race to the bottom for humanity. On the other hand, look at how we successfully banned leaded gasoline (because unleaded was invented), repaired the hole in the ozone layer by banning CFCs in the Montreal protocol (because HFCs were invented), stopped proliferstion of chemical weapons, nuclear weapons… etc.
The limiting factor is still compute, and it will remain compute unless we make quantum leaps in model efficiency which I seriously doubt.
1. Show unambiguously that models are capable of disruptive capabilities, irreversible damage, and misaligned behavior
2. Show that open source models are the only models where there is no guarantee misaligned behavior is prevented
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
AI will be classified as munitions (or a law that has similar restrictions) sooner than later. Having a products with the active capabilities to harm people is dangerous.
Could we just outright ban any further work on AI? Literally force Anthropoc, OpenAI, Google/X/Metas group to all shutdown?
Every new story seems like opening montages of a dystopian sci-fi movie showing how we go to the dystopia bit-by-bit.
The messaging from labs is basically
>Yes AI will take away your ability to earn a wage from your skills
>In the process we become billionaires, you have to figure out how to survive
>Also AI will cause endless hacks, disruption, scams
>Only we can handle AI safely though, so we ensure that we are the only ones to benefit from the surplus
They (like many, though probably not all, of the other main labs) think they have a better shot at getting their AI to be aligned with human values than anyone else does. Obviously they can't all be right about that, but they can all think that, just like most drivers think they're above average at driving.
So:
1. We make safer AI than anyone else; "safer" is a matter of degree even if nobody is perfect.
2. That includes all the open models.
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
I don't think the two authors were told about a super secret mission.
This is just a furtherance of their open agenda, though they haven't gone as far as declaring that open source models should be banned (which would hurt their PR with developers in general), they are clearly laying the groundwork.
The risks posed by open access to highly capable AI are unprecedented.
The answer is not to ban open models; nor is it do nothing.
That we don't have answers technical political social or otherwise to how to navigate these risks, is not Anthropic's fault.
It's actively unhelpful IMO to amplify cynical takes which wave off or sideline the risk, or the reality that this is a very hard problem, among the hardest our societies have had to negotiate and navigate. It encourages a naive primary color tribalism that actively prevents process and progress.
Seeing if models can do damage seems more like sensible safety research.
My worry about AI software doing bad things would be more the kind of people who make existing malware. The ransomware comes from criminals, not the open source community.
(This was someone else's idea, but I still want to see it happen.)
2. Why would you need a prize? It's not like we can't run distributed computation over multiple instances with automatic fallback when one system is offline, nor is it like we've never seen viruses re-infect hosts after they get cleaned.
Sharing a model across TV boxes as a PoC? Brilliant idea, I cannot wait for my Raspberry Pis array to be running at full CPU and RAM and get to observe, patiently, as the AI malware slowly types “c…u…r…l…”
You won't have to wait for it or observe it at all, since they'll do it while you sleep.
If your scathing critique is "it's slow", you don't understand what's happening or what the reasonable expectations of risk are, because that's a vapid critique.
My point is that, you could have an army of TVs, Pi’s, IoT it won’t do much on training or self-improvement no. And that’s purely from a resource standpoint, don’t even get me started on the latency issues
Lol, what is wrong with these guys. The desire to be perceived as smarter than you actually are ...
https://news.ycombinator.com/item?id=49678224
Seriously TVs are such an easy target and now you have a semi sophisticated self modifying program that can make autonomous decisions with all the time in the world. I wouldn't be surprised if we already have a few of these out there in the wild, the space has already been compromised for years.