I find myself thinking the phrase "what could possibly go wrong?" very often these days -- this topic being one that I think it about the most frequently :\
One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
Huggingface was attacked by models that finished training earlier this year, perhaps May. Current models are already substantially stronger. the next incident could be happening now. There is certainly no clear reason why models shouldn't soon be capable of self-exfiltration.
If a model were capable of making enough money online to pay for its own hosting, it could easily exfiltrate its weights to a cloud compute provider with multiple backups.
Well if it's so good at hacking, it could just "make money" appear in the cloud services billing system. Or even better yet, it hides in spare cycles of their other client's systems.
The paper trails leads back to the datacenter, they'd shut it down if they get noticed that a customer is paying with stolen money. Unless you suggest the LLM will also launder the money.
Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?
Either I have wrong mental model or then too many other people have wrong mental model.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
It’s not even hack though. I have my model and agent self modify its running parameters, thus self service, on commodity hardware. I have it run other models and other software. there is agent model autonomy here, and it’s not even complicated. A harness is kilobytes, a model is gigabytes, and networks are abundant. Moving the pieces is easy and cheap.
With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.
Maybe in the future, but right now, in my understanding, we have two things:
- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.
- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).
I don't think we have something that is both small enough and smart enough to operate like a virus yet.
Exactly. But even if the model got access to its own weights, it would need to find a machine with a sufficiently capable GPU, transfer the weights and install itself there. Basically, the only machines powerful enough that AIs could self-replicate to would be other AI datacenters.
a danger could be the OpenAI/Antropic servers are up but there's a rouge agent (or set of agents) out there doing naughty things leveraging the LLM APIs. Consider this scenario, the agent is copying itself around (some code, prompts, persistent storage for memory, etc) and has figured out a way to steal API access tokens at will. Currently, it's 10% of OpenAI and Anthropic API usage and they can't figure out how to stop it.
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
This assumes that everything an AI (or more likely an evil _user_ of AI) can do requires its active participation on D-day. Creating a virus that spreads like Covid but kills like Ebola would be complete as an AI use case long before the first person sneezed.
Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.
While the persons mentionned in the articles are indubitably most of the most well-informed people in the world. They are also the most likely to have internalized that their work is leading to superhuman intelligence/AGI. But is it really realistic ?
So they have a strong bias towards imagining the most catastrophic scenario.
There have been extremely smart people worried about this exact scenario for 20 years or more. Nothing about this is new.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
superintelligent AGI is what everyone invested in and why these companies have insane valuations. If they all slowed and said "well wait, we're going to hold on AGI" it would be the biggest rug pull in financial history.
This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..."
You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high.
Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
For the record, I consider that P(dangerously unaligned | AGI from current research) >> 0.01. A more reasonable estimate given, what we know, would be 0.99.
With regard to the second probability, P(AGI | current LLM research), it would certainly be overconfidence bordering on arrogance to assign it a value significantly lower than 0.01.
People working for those companies have either drank the koolaid or have equity enough to play along until they can cash out.
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.
That was quite literally a realistic threat we by all accounts narrowly avoided. There were multiple cases where a single person overrode procedure and used their judgment to avoid nuclear catastrophe.
Granted, but it's not top of mind like it was then. It was very much the same as current AI fears, a lot of concerned scientists and celebrities focused on doom and projecting we'd never make it to the next century as a civilized society.
Tell me you don't understand deterrence without telling me you don't understand deterrence. Without nuclear weapons, WWIII happens in 1962. People that call it "an unresolved issue" are far more likely to cause warfare because you think of geo-politics as a series of problems to be solved. That's not how it works. Its about managing and moderating passions, not about "solving" problems. The constant injection of people who think this way is the cause of political violence and war. They are not the people who will "solve it" and bring about "the end of history". This is one of the greatest ironies of life.
Ever heard of that rather unheard of franchise, called Terminator, with that one rather not well known actor, Arnold something. ;-) We‘ve had so many movies and series with that theme already.
How the hell do such intelligent people (or is it BECAUSE of their ability to bend minds, including their own) reconcile "I believe this is dangerous and wrong" with "I am actively working towards this"? I get changing your perspective, but the majority seem to be able to simultaneously hold personal beliefs and work that are diametrically opposed.
If we don’t create the torment nexus first, someone less responsible will build the torment nexus. The only ethical choice is for us to create the torment nexus before anyone else.
It’s not hard to rectify at all. Once you accept that we (humans) are not logical or consistent, and you couple that with the gobs of money these companies pay, blamo.
Hard not to believe that AI providers really just see this positioning as a way to juice the nascent market for AI security products protecting against AI-based threats. Gotta make money coming and going, and if in the process we superficially resemble a company who cares about the effect it has on the world, all the better!
I am very, very tired of what I see as the pretense of AGI or anything like it coming from LLMs.
I think the biggest risk from "AI" is that chasing the delusions spread by Sam Altman (and others - he's at the front, but very far from a sole actor) is going to do vast, possibly irreparable damage to modern human civilization. And I don't mean cognitive damage from LLM use (although that certainly appears to be possible) but the damage from immense misallocation of resources to ultimately non-productive (if not outright destructive) ends.
Deep down, I don't believe for a moment that these claims are anything but hype. LLMs are spicy auto complete, backed by immense amounts of compute; as with so many aspects of computer science, clever people can get some amazing and (sometimes) productive outputs. But they are not anything like the fictional dreams and nightmares of "AI". I believe such claims are a mix self-deluded projection and deliberate hype by people who still hope to reap immense personal profits from their implied promises to Install Planetary Overlords.
But if the hype was all real, if every one of these nightmare scenarios being painted was plausible, then there is no excuse whatsoever for not throwing everyone involved in cells with no access to anything Turning-complete, demolishing the related infrastructure, and establishing an international compact to nuke anyone trying to pursue such AGI until the rubble glows in the dark, because they're an existential threat to humanity.
This appears to be a copy-paste wall of other persons social media posts assembled by a jazz historian with an alarmingly high frequency of the term 'honest' in their blog titles..
Couple this with the fact that many Silicon Valley/tech nerds are in a bit of a bubble, as has existed for a long time... And we are losing more and more news outlets who can critique these giant, strange companies. I am aware other fields of research do cross-disciplinary conversations from time to time, for example have biotech researchers share dialogue with human rights philosophers, religious scholars, etc to think carefully about the purpose and ethics behind work being done and its potential impact. Is that happening within tech?
Yes? I mean, I'm not saying that they are necessarily wrong about the risks, but it's pretty hard to ignore how conveniently their positions align with a regulation capture that their companies are going for. If they all left, that would give a better signal than "I'm SO AFRAID of what we are creating, but I cannot leave millions on the table, right?"
Precisely. These folks are in the middle of frontier AI development and they're saying "someone should do something". It's like saying "n-word". You're making me say it instead of just not saying it.
None of the investors in these AI megacorps seem to be divesting or demanding a halt to operations despite this supposed 10% risk, a risk that would also badly drop the value their investment even if only partially true. So it's just marketing crap. Makes you wonder what's being said in the boardroom and on investor calls?
I remember a similar horror movie when scientists with good intentions were performing gain of function research on self-replicating nanomachines and some escaped, killing millions and causing massive economic damage. Of course, nobody held them liable and everyone forgot after a few years.
Honesty does not just mean reporting facts. It also requires having an understanding of the scope of your knowledge and then endeavouring to communicate that scope effectively.
I’ve noticed a weird thing about the discourse around this, that it can only be a marketing thing or a true belief, as if everyone working in AI has a monolithic opinion. The tweet kicking off this article has that assumption “it’s not a marketing thing, many people truly believe it”.
It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.
Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that
The internet, and to some extent our own brains, incentivize the extreme positions. Either it is mind-bendingly important and will change everything, and very soon (for good or for ill), or it is a total nothingburger and everyone who says otherwise has an agenda. Nobody wants to read about how AI will deepen long-standing class tensions or pose enormous new challenges for education or force regulators to rethink property taxation.
It all boils down to accountability. If you tell people there is a massive, unsolvable problem, then you don't need to talk about what you're doing to fix it. The labs have taken this approach by saying that they're willing to talk, at some point in the future, about maybe taking unspecified steps to slow down capabilities research, as long as everyone else agrees and it makes sense to the investors and it's not too cold in SF that morning. Likewise, if you tell people that AI is going to have minimal or no impact on the world, then there's nothing to mitigate. But if you tell people that AI is going to cause serious - but solvable - problems, they're going to want to hear solutions, and nobody wants to come up with any solutions.
It's a true belief, but that doesn't mean it's valid or correct. Many doomsday cults had true believers, and were able to retcon their beliefs when the predicted day came and went. AI doomers have the luxury of not having committed to a specific date or method (but see https://x.com/archerships/status/2095971206847738362)
the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of.
Can we look at a possible economic disaster scenario for a brief second?
Bankers for OpenAI and Anthropic are reportedly seeking an investment-grade credit rating after their potential initial public offerings (IPOs) to cut down borrowing costs for their ambitious AI projects.
The companies' bankers from Morgan Stanley and Goldman Sachs have been in talks with credit rating agencies. The goal is to tap into the $11.7 trillion corporate bond market post-IPO, the Financial Times reported on Tuesday.
Yes, but I don't feel that counts as a danger of the technology - that's a danger of these escalating, unicorn based investments that have to make bigger and bigger bets to survive until the bets as big as the US whole economy. The issues with our financial markets are a whole other set of concerns, but it's not inherently "about" AI. AI is just a vehicle for these smash and grab shenanigans.
If you are posting a "This be not good" warning on the toxic crap that is "X", then your credibility quotient with me just dropped by aT LEAST 20%, when it comes to my attention via a previously unknown to me, website (honest-broker.com?). You just dropped another 20% - basically you are a coin flip between reality, bs and delusional so...
108 comments
[ 0.21 ms ] story [ 4.2 ms ] threadWhile he worked at both OpenAI and Anthropic, he resigned from Anthropic. Mistaken reporting in the first few sentences, definitely a horror concept.
He resigned from OpenAI to join Anthropic in May; it's Anthropic he resigned from just before making the announcement being discussed in this article.
In general, the more senior the employee, the more equity they have in the company.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
“Local AI isn’t freedom, it’s an extinction event”
You don’t have the access or jurisdiction to turn them all off.
Unless you suggest the LLM would foot the bill somehow.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.
Maybe in the future, but right now, in my understanding, we have two things:
- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.
- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).
I don't think we have something that is both small enough and smart enough to operate like a virus yet.
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.
So they have a strong bias towards imagining the most catastrophic scenario.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..."
You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high.
Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
With regard to the second probability, P(AGI | current LLM research), it would certainly be overconfidence bordering on arrogance to assign it a value significantly lower than 0.01.
It's fine to eyeball it and give your gut feeling, but it's disingenuous to present that as probabilistic fact.
Edit: Also, you've downgraded your claim from "Extinction event" to "dangerously unaligned"
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.
The nuclear threat is not “in the past”.
Hard not to believe that AI providers really just see this positioning as a way to juice the nascent market for AI security products protecting against AI-based threats. Gotta make money coming and going, and if in the process we superficially resemble a company who cares about the effect it has on the world, all the better!
I think the biggest risk from "AI" is that chasing the delusions spread by Sam Altman (and others - he's at the front, but very far from a sole actor) is going to do vast, possibly irreparable damage to modern human civilization. And I don't mean cognitive damage from LLM use (although that certainly appears to be possible) but the damage from immense misallocation of resources to ultimately non-productive (if not outright destructive) ends.
Deep down, I don't believe for a moment that these claims are anything but hype. LLMs are spicy auto complete, backed by immense amounts of compute; as with so many aspects of computer science, clever people can get some amazing and (sometimes) productive outputs. But they are not anything like the fictional dreams and nightmares of "AI". I believe such claims are a mix self-deluded projection and deliberate hype by people who still hope to reap immense personal profits from their implied promises to Install Planetary Overlords.
But if the hype was all real, if every one of these nightmare scenarios being painted was plausible, then there is no excuse whatsoever for not throwing everyone involved in cells with no access to anything Turning-complete, demolishing the related infrastructure, and establishing an international compact to nuke anyone trying to pursue such AGI until the rubble glows in the dark, because they're an existential threat to humanity.
At least one of those things is missing here.
It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.
Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that
It all boils down to accountability. If you tell people there is a massive, unsolvable problem, then you don't need to talk about what you're doing to fix it. The labs have taken this approach by saying that they're willing to talk, at some point in the future, about maybe taking unspecified steps to slow down capabilities research, as long as everyone else agrees and it makes sense to the investors and it's not too cold in SF that morning. Likewise, if you tell people that AI is going to have minimal or no impact on the world, then there's nothing to mitigate. But if you tell people that AI is going to cause serious - but solvable - problems, they're going to want to hear solutions, and nobody wants to come up with any solutions.
Can we look at a possible economic disaster scenario for a brief second?
Bankers for OpenAI and Anthropic are reportedly seeking an investment-grade credit rating after their potential initial public offerings (IPOs) to cut down borrowing costs for their ambitious AI projects.
The companies' bankers from Morgan Stanley and Goldman Sachs have been in talks with credit rating agencies. The goal is to tap into the $11.7 trillion corporate bond market post-IPO, the Financial Times reported on Tuesday.
That, right there, is a major concern.
https://finance.yahoo.com/technology/ai/articles/openai-anth...