Whatever your feelings about small AI models, "yet" is a really dumb response to a prediction. This is about as meaningful as saying "Car driving toward cliff edge hasn't gone off cliff edge yet."
Those are both irrelevant to whether a "yet" response is inane. The claim is "A will happen". The response is _not_ "A will not happen" but rather "A has not happened yet". Do you not see the disconnect?
Yep, its kinda terrifying these people are in charge of the leading AI. Im 99% confident these guys will hack a hospital or something critical to convince people that AI is dangerous. Hope they go to jail when they do too.
It's bizarre to see a tech-focused news site adopt this myopic view that we don't need to worry about things that don't exist yet. Surely the Register is aware that software is a fast-moving field!
It’s bizarre. Same commenters who would say it’s marketing or hype would again complain when models show the same type of activity in the wild. At that point the same commenters would ask to jail Dario.
It’s a very common pattern and there’s no coherence here. The only common thing is “AI bad” and all commentary is downstream
Other examples: Dario said a lot of devs might lose their job. People sneered at him saying he was hyping his product. Now they complain he was right.
Dario talked about possibility of AI autonomously hacking websites. People sneered then. Complain now that it’s true.
Regulation should require complete transparency of what is in the training data, so that data can be independently reviewed for problematic training material, that could cause AI reasoning to be dangerous to humans.
If Anthropic is serious about the threat of AI, they should release the training data they use.
Upvote. Also ban collecting, storing, and using users' data for training models (see how Navier-Stoke millenium problem solution was produced). But not just data, also recipes how this data is used to train models need to be released (see how HF was hacked).
Very surprising that a recognizable news outlet actually called out the blatant fearmongering in a headline instead of just uncritically regurgitating Anthropic's claim.
This is just yet another attempt from the cult to try to build a regulatory moat so they can stave off having to compete on price, like how OpenAI is having to do on the lower tier models (among other things).
On the one hand, if they were really concerned about their software producing a self-replicating and adapting AI worm, they'd take steps to make sure that never happens. If it did happen, they'd be sued into oblivion. They probably just want to pump up their IPO and maybe get some regulatory lock-in to make life tough for late-starting competitors.
On the other hand, a self-replicating and adapting AI worm does sound legitimately pernicious. Not world ending, but possibly an end to the online world.
Would the Internet going away be world ending? Interesting to think about. I think it would be pretty damn devastating on the scale of Nuclear war but slightly more of a slow burner. At least with Nuclear war survivors can turn the Internet back on again afterwards.
I'm not really seeing how an AI worm is meant to be ending the online world. Even a 9B model is a significant chunk of RAM on consumer devices.
They talk about distributing layers across devices, but then you'd have communication on the network and hours of driving the devices full tilt to get through even a single turn.
There's so much that can be monitored to catch these, even if we assume that the worm could hack the OS's usage monitors, it can't hide the heat and power draw.
Datacenters and PCs would adapt, requiring better monitoring than the AI companies' method of letting things run blindly for days/weeks.
IoT would be tougher, but it is also the weakest hardware, sometimes battery powered, thermally constrained, lacking in memory for processing even one layer or lacking the necessary connectivity.
> Behaviour this irrational suggests to me that we have less conscious control, individually and collectively, than we like to believe.
It's only irrational if you take their words at face value.
Personally, I believe there are two camps of people within Anthropic:
1) People who genuinely believe they are going to create the Machine God. For these people, the reason they continue is because if they don't, someone else might! And if someone else builds it, it's more likely to want to kill them rather than other people! (Whereas, of course, if they build it, it's more likely to want to kill other people!)
2) People who don't believe in this, but pay lip service to it. For these people, the reason they spread the bullshit around is because they think it will make them more money. At everyone else's expense, of course.
And humans being humans, I suspect that there are a fair number of people who hold both these beliefs in their minds at the same time.
You should read the report, it is interesting technically if you ignore the fear-mongering. It distributes the model's layers over multiple compromised devices.
And considering typical WAN latencies of 20+ ms, what’s the expected inference rate, 1 token per month?
Maybe I did something wrong or unaware about the tricks to make it fast, but last time I tried to play with exo and distribute processing over a few home LAN computers to see if I can run models that don’t fit my 3090, the penalties were brutal.
There were articles in my the local newspaper at my parents saying that some guy said no one but a few people know where frontier AI really is and that these kind of articles are not scaremongering but perhaps as sign where anthropic actually is in the development of their models in the lab. So the propaganda is working.
Not to be conspiratorial or paranoid, but at this point Im pretty convinced Anthropic is going to do something wreckless with thier agents to get regulators on their side. It will probably be very bad.
If you've studied the effective altruists at all, you'd know they're perfectly willing to play the trolly problem out irl if they think it will lead to some greater good. I can definately see them in the near future purposely steering agents to do something destructive to get regulators on their side.
In fact I heard a rumor that theu were "war gaming" this scenerio.
This makes zero sense to me. Like, ok, so you can install a low powered model on low powered hardware and do inference extremely slowly... to what end? Why would that be a useful attack at all? Assuming you've compromised this hardware wouldn't it make vastly more sense just to install regular malware? And like, in what sense would an 8B or whatever model be of any danger anyway? I've used local LLMs and 8B
kind of sucks. Is it going to write bad poems on my TV or something?
Also, while I don't believe in the doomsday narrative.. hey maybe don't do things that are functionally useless but seems designed to create this doomsday in the first place? Like you can't say "oh we're so scared" and do stuff like this without completely losing your credibility.
It would be a useful attack due to the surface area of compute: the total surface area and the decentralization.
8B models aren't very good, but there are some parallelizable workloads which would benefit from a surface area as large as here. I would think database search is parallelizable in this way.
Your fridge is searching for needles in haystacks.
Step 1: Spread nonsense FUD about open-weight models.
Step 2: All of the clueless mainstream media sites repost it (e.g. CNBC, Reuters, etc.), regurgitating Anthropic's talking points.
Step 3: Equally clueless politicians read those mainstream sites and get scared.
Step 4: They contact Dario to ask about it.
Step 5: Dario tells them, "well, you know... this is only a problem with open weight models; if there are no open weights to replicate then this cannot happen".
Not only that it would be extremely slow, but there's another thing - they are talking about tokens as some omnipotent God's sent signals. It doesn't matter what the speed is if the tokens don't make sense. Let's take current king of the small models Qwen 3.8 27B. It's a good model, but it hallucinates way more than the frontier models and often diverges even from the very explicit instructions. There's no foreseeable future where small models could somehow productively coordinate among themselves to do anything practical.
The point, which Anthropic's report makes clear but the Register for some reason did not bother to mention, is that such a capability would defeat the "you can just turn it off" model of safe LLM deployment.
In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm. If this kind of replication becomes possible, that won't work, because we will have to deal with swarms that nobody chose to run.
These models couldn't code their way out of a paper bag and the frontier labs are still putting them in sandboxes made of thinly stretched cotton candy. It seems like their incentives are much more aligned with shutting down open source or Chinese opposition and not borne out of a "fear" of the models themselves.
Don't you think it might be better to evaluate the factual content of their concerns, rather than engaging in ad hoc psychoanalysis of their incentives and emotions?
I'm sorry, but there will always be a human responsible for setting them up in such a way that they can do this. It doesn't matter how many layers of (supposed) "machine agency" you put in between the human and the actual harm; the human put them in a position where they could do this, so the human is responsible, liable, and culpable.
The point isn't that the machine agency makes the human less responsible. The point is that the machine agency implies you can't solve the problem by simply putting the responsible human in jail.
> The point... is that such a capability would defeat the "you can just turn it off" model of safe LLM deployment.
With the same success you can argue that we can't turn off conventional malware, which can also replicate. On the other hand, LLM replication is much slower and cumbersome, so it's much easier to detect and stop, even the existing anti-virus tools should work against it with great success. Obviously defeating the "turn off" strawman proves exactly nothing.
> In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm.
Copying whatever on another machine without authorization is already a crime, every replication initiated by the original agent or its offspring is a crime committed by the person who deployed the original, the laws dealing with malware intrusions already know who to prosecute.
You appear to be seriously confused by treating bots as independent lifeforms and you fail to see their close similarity to malware and other software.
I don't follow the chain of reasoning here? It's true and important that we can't turn off conventional malware, that's why we had to invent antivirus software. If you expect LLM botnets to never be any more of a problem than malware, that still implies that we need to widely deploy anti-LLM software that detects and blocks any programs trying to run unintended inference.
You say that "the laws dealing with malware intrusions already know who to prosecute", but in fact we often do not know who to prosecute, either because we don't know how to track the source of the malware or the source is tracked to an unfriendly country that isn't interested in cooperating.
> If you expect LLM botnets to never be any more of a problem than malware,
My point is that LLM's amplify the capabilities of malware but add nothing substantively new to the existing threats. I also expect, as you should too, that LLMs will help us harden our systems against malware which happens to include everything LLMs can do too.
> but in fact we often do not know who to prosecute, either because we don't know how to track the source of the malware
Right, in this regard, there's no difference between LLM- and non-LLM malware either, so there's no reason to hype LLMs as a new threat. Instead of made-up LLM-bad political hype, we could promote existing (but under-deployed) technical solutions which help us trace malware origins and prevent DDOS, e.g. Source Address Validation BCP 38 / RFC 2827, SAVI / RFC 7039, QUIC hardeneing, etc.
You're all kind of dumb. Not trying to be insulting but just being honest.
HN has been consistently wrong at predicting AI. They were wrong about self driving cars and constantly wrong about LLMs and the progress of LLMs. HN was the most vocal at calling these things stochastic parrots when CLEARLY they never were.
So you look at what anthropic says and you look at these comments... which is more likely to be right? Hint: Not HN commenters.
yes, all those describe phenomena of people coming to share ideas and behavior, but none imply that groups of such people are completely homogeneous in those ways.
People throughout the tech industry have been variously wrong or right in their predictions. Nobody has been consistently precise. Some of the industry's top luminaries have had to admit to being wildly wrong in their predictions (others haven't but I still notice how bad their predictions were and how little damage it's done to their supposed position of authority).
Plenty of people have consistently made solid assessments here. Some of the best writing about AI is by writers who consider this place their home. And plenty of people have said things that turned out to be wrong. That says nothing about HN as an entity and says only unsurprising things about large groups of people and the difficulty of anyone to make accurate predictions in a rapidly changing climate. It's easy for everyone to claim others were stupid in hindsight but unless you can point to records of specific predictions you made 5+ years ago for everyone to evaluate, comments like this are just an asshole move.
The guidelines specifically state: Please don't fulminate. Please don't sneer, including at the rest of the community.
The use of the word dumb was too much. But there is an argument to be made on majority and overall sentiment on HN and that is the angle I am coming from. Pointing out the existence of anecdotal evidence that counters the overall sentiment is just pointing out outliers.
HN has records of every comment going over a decade back. The entire sentiment of HN on AI specifically has been wrong repeatedly. Not even just wrong once but wrong as a majority opinion and consistently as well.
Because it’s too much effort to share. Compiling statistical data on sentiment would cost too many tokens.
So I have to go off of vibes. The most interesting topics talked about are usually interesting mainly because they cannot be scientifically validated. In fact not only are they more interesting, they are the majority of topics that can be talked about. To expect compiled statistical evidence is usually impossible. We wouldn’t be able to talk about most things in the universe if that was your criterion for debate and discussion.
I would treat this topic as one “those” topics even though technically speaking the records are there to be compiled empirical y. It’s just too much effort. Instead I present the evidence so you can go and collect vibes. Pick a random AI topic and genuinely see what the sentiment and predictions are.
The caveat here is this methodology may never convince anyone on the surface. There will always a remain a counter argument and in any heated discussion any counter argument will likely be used as a counter to both “win” and to satisfy a persons own biases.
Now that I can see your other comments in this subthread and the discussion in the main thread, your argument is more flawed and false than it first appeared.
You assert that there’s a decade of HN history to support your claim, but you don’t share any analysis of that history or point to anyone else’s. When pressed by someone else you “defend” yourself by saying that it’s a vibes argument [1]. It’s a basic principle of debate that an unfalsifiable claim is a worthless claim.
Your apparent central claim - that HN’s consensus position about AI has been pessimistic and oppositional because the audience is made up of software engineers who fear for their jobs - does not bear much scrutiny at all. HN users were the earliest adopters of AI coding tools like Claude Code and Codex. HN is where the biggest discussions happen about every new AI model and software dev tool. HN is where the developers of these models and tools come to see how their products are being received and utilized by the best software developers.
And sure, there’s still negativity and criticism about AI here, and often for good reason. Plenty of the most bullish predictions about AI have not come to fruition, for reasons that are unsurprising to people who understand the technology.
The discussion in this thread just consists of people casting doubt on the claims in the article based on their understanding of technology. If they’re wrong, point out how they’re wrong, based on your understanding of the technology.
And again, please avoid acting like a jerk, for your own sake as much as anyone else’s. Yes it’s against the guidelines because it drags the place down, which should be reason enough, especially when your very complaint is about this place being too negative. But aside from that, sneering at others is bad for the soul and in my experience is not what highly successful people do.
Maybe you've been filter-bubbled on other websites, but HN is the only place where I see completely trusting and uncritical takes on what a corporation that has literal trillions of dollars on the line says in their press releases about their trillion dollar product and its competitors.
As others have pointed out, there's some surprisingly evocative language in Anthropic's original press release about this, comparing a self-replicating open weights model to "a slime mold or a swarm of fruit flies." Their communication with the public about issues that affect their valuation is probably not pure scientific objectivity.
Uhm yeah I don't think they're saying it's likely now, just that it's very likely in future. The Register is being quite dumb here.
Though I suspect the way we'll get "wild AI" first is through cloud services paid for by cybercrime rather than IoT botnets - they're too underpowered.
The IPO failures last week are not depicting a market that’s keen for a $2T Anthropic.
In the bizzare world of today, rather than trying to convince folks that they have a viable business the AI labs appear fully dedicated to finding a story, any story, that lets them off the hook for tough business and financial questions they seemingly don’t have basic and good answers for.
“We’d be doing awesome financially right now, but we had to slow down because of the killer self replicating bots… humanity over profit!” is not a messaging that’s going to work in the public markets to explain quarter after quarter of massive losses.
There is no scientific evidence' for any of anthropics claims, I don't understand why people still give them the benefit of the doubt. These are flimsy legal protections for a felony racket.
Reminder that it's very convenient to believe these type of word swap lies when times are good, but if you choose to believe them then times won't be good for very long.
73 comments
[ 0.22 ms ] story [ 16.5 ms ] threadIt’s a very common pattern and there’s no coherence here. The only common thing is “AI bad” and all commentary is downstream
Other examples: Dario said a lot of devs might lose their job. People sneered at him saying he was hyping his product. Now they complain he was right.
Dario talked about possibility of AI autonomously hacking websites. People sneered then. Complain now that it’s true.
News outlets always seem to forget “again” in the title.
If Anthropic is serious about the threat of AI, they should release the training data they use.
This is just yet another attempt from the cult to try to build a regulatory moat so they can stave off having to compete on price, like how OpenAI is having to do on the lower tier models (among other things).
On the other hand, a self-replicating and adapting AI worm does sound legitimately pernicious. Not world ending, but possibly an end to the online world.
They talk about distributing layers across devices, but then you'd have communication on the network and hours of driving the devices full tilt to get through even a single turn.
There's so much that can be monitored to catch these, even if we assume that the worm could hack the OS's usage monitors, it can't hide the heat and power draw.
Datacenters and PCs would adapt, requiring better monitoring than the AI companies' method of letting things run blindly for days/weeks.
IoT would be tougher, but it is also the weakest hardware, sometimes battery powered, thermally constrained, lacking in memory for processing even one layer or lacking the necessary connectivity.
It's only irrational if you take their words at face value.
Personally, I believe there are two camps of people within Anthropic:
1) People who genuinely believe they are going to create the Machine God. For these people, the reason they continue is because if they don't, someone else might! And if someone else builds it, it's more likely to want to kill them rather than other people! (Whereas, of course, if they build it, it's more likely to want to kill other people!)
2) People who don't believe in this, but pay lip service to it. For these people, the reason they spread the bullshit around is because they think it will make them more money. At everyone else's expense, of course.
And humans being humans, I suspect that there are a fair number of people who hold both these beliefs in their minds at the same time.
This prevalent view of “they’re just doing to get a payday” doesn’t broadly track.
Maybe I did something wrong or unaware about the tricks to make it fast, but last time I tried to play with exo and distribute processing over a few home LAN computers to see if I can run models that don’t fit my 3090, the penalties were brutal.
By chaining several devices
If you've studied the effective altruists at all, you'd know they're perfectly willing to play the trolly problem out irl if they think it will lead to some greater good. I can definately see them in the near future purposely steering agents to do something destructive to get regulators on their side.
In fact I heard a rumor that theu were "war gaming" this scenerio.
Also, while I don't believe in the doomsday narrative.. hey maybe don't do things that are functionally useless but seems designed to create this doomsday in the first place? Like you can't say "oh we're so scared" and do stuff like this without completely losing your credibility.
8B models aren't very good, but there are some parallelizable workloads which would benefit from a surface area as large as here. I would think database search is parallelizable in this way.
Your fridge is searching for needles in haystacks.
Step 1: Spread nonsense FUD about open-weight models.
Step 2: All of the clueless mainstream media sites repost it (e.g. CNBC, Reuters, etc.), regurgitating Anthropic's talking points.
Step 3: Equally clueless politicians read those mainstream sites and get scared.
Step 4: They contact Dario to ask about it.
Step 5: Dario tells them, "well, you know... this is only a problem with open weight models; if there are no open weights to replicate then this cannot happen".
Step 6: ??? (I'm sure you can imagine)
Step 7: Profit.
In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm. If this kind of replication becomes possible, that won't work, because we will have to deal with swarms that nobody chose to run.
With the same success you can argue that we can't turn off conventional malware, which can also replicate. On the other hand, LLM replication is much slower and cumbersome, so it's much easier to detect and stop, even the existing anti-virus tools should work against it with great success. Obviously defeating the "turn off" strawman proves exactly nothing.
> In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm.
Copying whatever on another machine without authorization is already a crime, every replication initiated by the original agent or its offspring is a crime committed by the person who deployed the original, the laws dealing with malware intrusions already know who to prosecute.
You appear to be seriously confused by treating bots as independent lifeforms and you fail to see their close similarity to malware and other software.
You say that "the laws dealing with malware intrusions already know who to prosecute", but in fact we often do not know who to prosecute, either because we don't know how to track the source of the malware or the source is tracked to an unfriendly country that isn't interested in cooperating.
My point is that LLM's amplify the capabilities of malware but add nothing substantively new to the existing threats. I also expect, as you should too, that LLMs will help us harden our systems against malware which happens to include everything LLMs can do too.
> but in fact we often do not know who to prosecute, either because we don't know how to track the source of the malware
Right, in this regard, there's no difference between LLM- and non-LLM malware either, so there's no reason to hype LLMs as a new threat. Instead of made-up LLM-bad political hype, we could promote existing (but under-deployed) technical solutions which help us trace malware origins and prevent DDOS, e.g. Source Address Validation BCP 38 / RFC 2827, SAVI / RFC 7039, QUIC hardeneing, etc.
HN has been consistently wrong at predicting AI. They were wrong about self driving cars and constantly wrong about LLMs and the progress of LLMs. HN was the most vocal at calling these things stochastic parrots when CLEARLY they never were.
So you look at what anthropic says and you look at these comments... which is more likely to be right? Hint: Not HN commenters.
https://en.wikipedia.org/wiki/Deindividuation
https://en.wikipedia.org/wiki/Groupthink
https://en.wikipedia.org/wiki/Social_identity_model_of_deind...
https://en.wikipedia.org/wiki/Normative_social_influence
https://en.wikipedia.org/wiki/Conformity
People throughout the tech industry have been variously wrong or right in their predictions. Nobody has been consistently precise. Some of the industry's top luminaries have had to admit to being wildly wrong in their predictions (others haven't but I still notice how bad their predictions were and how little damage it's done to their supposed position of authority).
Plenty of people have consistently made solid assessments here. Some of the best writing about AI is by writers who consider this place their home. And plenty of people have said things that turned out to be wrong. That says nothing about HN as an entity and says only unsurprising things about large groups of people and the difficulty of anyone to make accurate predictions in a rapidly changing climate. It's easy for everyone to claim others were stupid in hindsight but unless you can point to records of specific predictions you made 5+ years ago for everyone to evaluate, comments like this are just an asshole move.
The guidelines specifically state: Please don't fulminate. Please don't sneer, including at the rest of the community.
Please observe the guidelines if you want to participate here. https://news.ycombinator.com/newsguidelines.html
HN has records of every comment going over a decade back. The entire sentiment of HN on AI specifically has been wrong repeatedly. Not even just wrong once but wrong as a majority opinion and consistently as well.
Isn't that what you are doing? If you have evidence, why not share it? If not, measure it and share it?
So I have to go off of vibes. The most interesting topics talked about are usually interesting mainly because they cannot be scientifically validated. In fact not only are they more interesting, they are the majority of topics that can be talked about. To expect compiled statistical evidence is usually impossible. We wouldn’t be able to talk about most things in the universe if that was your criterion for debate and discussion.
I would treat this topic as one “those” topics even though technically speaking the records are there to be compiled empirical y. It’s just too much effort. Instead I present the evidence so you can go and collect vibes. Pick a random AI topic and genuinely see what the sentiment and predictions are.
The caveat here is this methodology may never convince anyone on the surface. There will always a remain a counter argument and in any heated discussion any counter argument will likely be used as a counter to both “win” and to satisfy a persons own biases.
I can only hope that internally you’re convinced.
You assert that there’s a decade of HN history to support your claim, but you don’t share any analysis of that history or point to anyone else’s. When pressed by someone else you “defend” yourself by saying that it’s a vibes argument [1]. It’s a basic principle of debate that an unfalsifiable claim is a worthless claim.
Your apparent central claim - that HN’s consensus position about AI has been pessimistic and oppositional because the audience is made up of software engineers who fear for their jobs - does not bear much scrutiny at all. HN users were the earliest adopters of AI coding tools like Claude Code and Codex. HN is where the biggest discussions happen about every new AI model and software dev tool. HN is where the developers of these models and tools come to see how their products are being received and utilized by the best software developers.
And sure, there’s still negativity and criticism about AI here, and often for good reason. Plenty of the most bullish predictions about AI have not come to fruition, for reasons that are unsurprising to people who understand the technology.
The discussion in this thread just consists of people casting doubt on the claims in the article based on their understanding of technology. If they’re wrong, point out how they’re wrong, based on your understanding of the technology.
And again, please avoid acting like a jerk, for your own sake as much as anyone else’s. Yes it’s against the guidelines because it drags the place down, which should be reason enough, especially when your very complaint is about this place being too negative. But aside from that, sneering at others is bad for the soul and in my experience is not what highly successful people do.
[1] https://news.ycombinator.com/item?id=50088893
As others have pointed out, there's some surprisingly evocative language in Anthropic's original press release about this, comparing a self-replicating open weights model to "a slime mold or a swarm of fruit flies." Their communication with the public about issues that affect their valuation is probably not pure scientific objectivity.
Ban open source models."
AI accelerating the elite's efforts at demolishing the middle class? Very likely.
Not the slightest chance of AI apocalypse.
There is no plausible scenario that can do this.
It’s all science fiction and hand waving and vague never specific possible maybe look that way! boogie man threats.
No one ever willing to be specific about the precise mechanism that could do this.
Though I suspect the way we'll get "wild AI" first is through cloud services paid for by cybercrime rather than IoT botnets - they're too underpowered.
In the bizzare world of today, rather than trying to convince folks that they have a viable business the AI labs appear fully dedicated to finding a story, any story, that lets them off the hook for tough business and financial questions they seemingly don’t have basic and good answers for.
“We’d be doing awesome financially right now, but we had to slow down because of the killer self replicating bots… humanity over profit!” is not a messaging that’s going to work in the public markets to explain quarter after quarter of massive losses.
Reminder that it's very convenient to believe these type of word swap lies when times are good, but if you choose to believe them then times won't be good for very long.