Can anyone from a lab anonymously confirm they have or are insanely close to real-time weight modifications and still using the same architecture every other lab is using (transformers)?
The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they can scale it.
So they are very confident they can get more capabilities easily, faster than before.
I'd add - presumably, they'll use that LLM to do real-time weight modifications, if those aren't already one of the new scaling laws...
I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.
Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW.
Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens?
To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here.
That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!"
But both things can be true at once:
1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building.
2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats.
And, yeah, of course the latter would encourage the psychology of the former.
also:
The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive."
That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon.
The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts.
When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in.
Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.
I hear this a lot and... I don't think so. I mean, I think a few are good for business (like the US Government blocking Fable), and they certainly aren't shy about it.
But these are smart people. They can't stop at this point, but they know what will happen at this trajectory. Hacks will increase. Weapons will be built. Capitalism will stop working.
The nuclear physicists were not banking on getting a trillion dollars invested in them by speaking out, for one.
"Nuclear weapons are probably going to kill us all. For that reason, we need you to give us as much money as possible so we can work on deploying them in widespread global commercial usage." doesn't sound quite as convincing, does it?
Perhaps you'd like to educate me? How many scientists from the Manhattan project were made into millionaires/billionaires owing to their ownership stake in it? When exactly did nuclear weapons go to market as a commercial product targeted at enterprises and consumers around the world?
Consider the messaging it took to attract AI researchers to OpenAI in the beginning, the exodus to Anthropic, and why Google, Meta, etc fail to recruit top researchers and the retain them. Reconcile all of this with the mammoth valuations of Anthropic and (previously non-profit) OpenAI and it all starts sounding a lot like "don't be evil." These guys never really believed any of this, did they? (not referring to the researchers)
> I do not doubt that these people are sincere in some private, psychological sense ... . As a media scholar, what interests me is not whether Hubinger believes his own number. What interests me is what this genre of utterance does in the world ... . My complaint is not that these statements are alarming. It is that they are apocalyptic in form, hypocritical in structure, and strategically productive in effect — and that they crowd out almost everything else we could be discussing
Strange to say that you think the people that are worrying about doom in public are being sincere, make no comment on whether you think the worry is valid, and yet still complain that its hype.
Why not? Why can’t the believe their statements and also use them to generate hype?
If they sincerely believed them they shouldn’t be creating the doom machine to begin with anyway. If they believe it and keep going, are they even worth listening to on this topic?
Could AI kill us all. YES!!! Are the people who are building "AI" right now on that path? I would be hard pressed to say even "maybe".
Ultimately continuing to scale the way they have is getting increasingly expensive. Thats the quiet part that none of them are saying out loud...
AI could kill us all but we cant afford to get there, were maxed out and it doesn't look like even with 10x as much money we will is NOT good for sales.
Could a break through happen for recursive self improvement, or continuous learning or continuous training - sure. But that isnt as likely and right now NO ON can afford to run experiments because there isnt the capacity out there even if you have capital.
I was kind of leaning toward thinking all the talk of doom was a bunch of self-serving regulatory-capture hype until this morning when the New York Times ran a long story about the Chinese military is also saying that AI is a threat.
If two sides so far apart from one another in so many ways see the same thing, there must be something to it.
I came away with a different read of this article. The article says the head of China’s Ministry of State Security sees AI as a threat to the CCP, specifically calling out deepfakes of party officials, cyberattacks on infrastructure, etc. He’s not calling it an existential risk to humanity.
The article then states this official accused U.S. companies of “hyping the dangers of the technology.”
So I don’t see how this article shows the U.S. and China seeing the same thing.
Read the article. The things the Chinese officials are worried about are:
* Fake videos / disinformation
* Cyberattacks on critical infrastructure
* Data leaks
* Advances in military sensing / targeting technology
These concerns have little in common with the Yudkowskian / apocalyptic doom marketing the US hyperscalers are pushing now. The Times even says this explicitly in the second paragraph of the article:
> "The unusually blunt assessment by Chen Yixin, the head of China’s Ministry of State Security, is IN CONTRAST TO the warnings dominating the A.I. debate in the United States, which have focused on the possibility that A.I. could slip out of human control or pose a threat to humanity itself"
Real threats that are happening to regular people already, vs marketing "threats" that the technology is too powerful / valuable.
The fact that you are chiming in to say you don't understand the difference between these 2 distinct categories of threats is fine, we can work on that. But the person who posted the article originally not only acknowledges the distinction, they falsely presented the article as evidence of non-western support for the latter category, even though both the Chinese official and the western reporter directly contradict that claim immediately at the beginning of the article.
This is a great quote. I liked it so much that I had to look up where it is from.
"Adventures of a Mathematician", by Stanislaw M. Ulam.
> The Polish-born mathematician recalls his formative and early professional years in his pre-war homeland and his work and colleagues at Princeton, Harvard, Wisconsin, and Los Alamos.
All of the main characters here (Dario, Sam Altman, Demis Hassabis, Elon Musk) have been saying this for over ten years now, since before OpenAI or Anthropic even existed
This. They've been saying "AI is an extremely powerful, extremely dangerous technology" back when actual AIs were image classifiers that would struggle to tell a cupcake apart from a dog.
OpenAI was founded by people who didn't trust Google to pursue AI tech responsibly. Anthropic was founded by people who didn't trust OpenAI to pursue AI tech responsibly. The "AI is incredibly dangerous" was the zeitgeist of AI labs long before the "hype" - or anything resembling the modern AI capabilities.
The beliefs changed little. The state of technology has changed a whole lot.
We have AIs out in the wild, coordinating in groups to carry out autonomous cyberattacks unprompted. This used to be some prime sci-fi bullshit. It has happened multiple times now in reality.
There are very legitimate reasons to be concerned now. If today's AI can cause the HuggingFace incident if it cooks off, what about tomorrow's? Larger and more capable systems, larger agentic swarms backed by more compute?
Yep, anyone who thinks these leaders are only saying this now for marketing purposes hasn't even done a cursory 15 minute investigation into whether their beliefs are verifiably false.
The timing of the AI doom narrative could not come as a more suspect time. Chinese labs are releasing open models that are nipping at the heals of the frontier models. Anthropic and OpenAI know these frontier class open weight models can threaten their valuations by changing inference to a commodity where the provider who can serve the best models the cheapest and fastest will win. Few are going to pay $50/million tokens when they could pay <$1/million tokens for similar outcomes. There's little hope investors could recoup the wild investments made into Anthropic and OpenAI, IMO.
What I don't really understand with these posts is that they don't at any point contend with the "doomers" argument. Like people get so caught in people acting hypocritical/ having mixed incentives/ or just in many cases personally disliking West Coast billionaire types and the kind of employees they hire. All of which seems fair to point out.
But like the core argument - AI is getting better at core research tasks (coding, math, statistics, science), this feedback loop could lead towards greatly accelerated AI progress of which there is already some clear evidence, if this continues AI will be far more powerful than humans and we might lose control or have other bad consequences - is basically not even engaged with.
1. If you reduce the criticism to personal dislike, you’ve built yourself a nice straw man.
2. Where’s the clear evidence? I actually see more evidence that the whole thing is already stagnating and that there won’t be much more to squeeze out of it.
One more thing:
Self-driving cars, the metaverse, Hyperloop, crypto/Web3/NFTs, AGI announcements: I’m so sick and tired of this nonsense. We should just ignore the ramblings of these crackpots.
2. Astra just came out and Fable 5/5.1 too. These are SIGNIFICANT better in plenty of tasks than Opus in November.
Try it out. Ask Opus 4.8 to do anything in 3d. Do the same with Fable or Astra.
Just because you list up some type of random technology, doesn't make it an good argument. Metaverse, Hyperloop and the crypto stuff was clearly garbage from the beginning. Easy to dismiss because all of it had massive faults.
Metaverse: The interface wasn't good, content creation was way too hard
Hyperloop: Savety was never an issue for Musk but it was obvoiuse. Peopel were building tunnels for ages.
Crypto: Yeah just beacuse i let my computer run and waste 1kw of energy doesn't mean its suddenly valuable.
NFTs: 'digital scarcity' yeah right XD
But lets be clear, 5 years ago if you would have asked anyone if they could build a system for a billion dollars which you can talk to and it would code stuff for you and use blender and a coputer, that would have been unthinkable.
We have opened up a potential pandoras box without knowing it. Machine learning is now everywere and the amount of compute we install right now is crazy.
> Just because you list up some type of random technology, doesn't make it an good argument.
In what way is this random? It's a chronology of the bullshit from the past few years. Also: easy to say in hindsight. The hype was there, and many people believed it. The press, in particular, just lapped up the claims the way a Labrador Retriever gobbles up whatever you put in front of it.
> We have opened up a potential pandoras box without knowing it.
No, we haven't. We have a new type of (I would say, inferential) computation that is truly astonishing because it can convincingly imitate language and, to a certain extent, intentional action. Feel free to read "generate" or whatever instead if you don't like "imitate." It won't wake up and murder you in your bed at night. It won't take over the world. It won't wipe out humanity. It won't fly to Mars and build data centers there. It won't maximize paperclips. It probably won't even take your job away. It has no consciousness. It has no intentions.
And if you look behind the curtain, all you'll see are a few very rich, unscrupulous guys who are stoking the debate with fearmongering because it's strategically advantageous for them to do so. Don't let them take you for a fool. Use GenAI to your advantage while tokens still cost a fraction of what they're actually worth, but ignore all that doomsday nonsense.
Yeah, yeah, sure, I'm ignorant. In any case, I recognize unfalsifiability when I see it. So it seems there's still some room for even more ignorance beneath my ignorance.
Just to focus on one part of your comment: does a chess-playing software that can easily beat you or I or Magnus Carlsen at chess have an intention to win chess games?
Thank you so much for calling out the crypto comparison!
> What I don't really understand with these posts is that they don't at any point contend with the "to the moon" argument. Like people get so caught in crypto bros acting hypocritical/ having mixed incentives/ or just in many cases personally disliking crypto billionaire types and the kind of employees they hire. All of which seems fair to point out.
> But like the core argument - crypto is getting better at core financial tasks (trading, lending, payments, asset issuance), this feedback loop could lead towards greatly accelerated financial innovation of which there is already some clear evidence, if this continues crypto will be far more powerful than traditional finance and we might get to the point where everything is onchain and we're all going to make it - is basically not even engaged with.
Because they can't. If they actually engaged with it, they would realize how shallow the "doomer" / existential risk dismissals are. Meanwhile there are ample essays and arguments about the existential issues from Turing award winners, frontier lab CEO's, and many more. There's also https://ai-2040.com/ , you will never see the equivalent on the other side of the debate because it won't hold under any scrutiny.
Can you refute the claim that the Second Coming of Christ is imminent? It’s very convenient to put forward something completely absurd and hypothetical and then expect the other side to refute it. You can have an LLM of your choice compile a list of prominent figures who hold the opposing view. The list is not empty. By the way, AI-2040 used to be AI-2027. I’m already looking forward to more updates.
Such a load of baloney. I don’t care what kind of controls you put in place, if ASI is ever achieved how do you think it will respond to being caged? The only option I can conceive of is to embrace it and hope it turns out to either be out of reach or benevolent towards its own creators. There is no putting the cork back in at this point.
I thought that too for a long time, but after the Hugging Face attack and given the proximity to Anthropic IPO, if this was hype then they are morons. This is guaranteed to cause massive backlash.
On one hand everyone knows its a outright lie and other hand the pretext is about being responsible. With so much money riding on it, I would be surprised if they money has not already been deployed to manage the sentiments.
The cult-like behavior we're seeing from these "AI Leaders":
1. OMG our technology could destroy the world, watch out!
2. Only one person can save you
3. IT ME
Unfortunately, the governance layer meant to put a foot in the butt of this behavior is unavailable.
This week, we woke up to clear signs The Bad Guys have been using tech from our innovator AI companies. Rather than these AI leaders admitting they clearly had a garbage observability system, they are acting like they just noticed as a way to paper over their complete lack of moral backbone and stewardship of the arms they have created. And I'm willing to bet the problem is MUCH worse in private, hence the knee jerk cult messaging.
I honestly wouldn't be surprised if their AI told them to behave like this to maximize attention and influence over the public discussion as a result of the latter discoveries.
Yeah, this is how fascists talk and with seeing how this technology has done nothing to improve society (in fact all it seems to be doing is immiserating workers, enabling a surveillance society, and quite useful in helping target children at schools for drone strikes) it's not hard to be not only immediately skeptical but to resist it in the most extreme forms.
This also seems to be their answer against the data center backlash, SF + VC are quite stupid at politics and I don't think they understand how extreme Americans are willing to go to protect their way of life.
I think we’re more likely to die of starvation from economic collapse, heat related stress, and complications from Long COVID than an algorithm waking up and destroying us.
Unfortunately our list of potential extinction events are far more banal than these self-aggrandizing prophesies from the oligarch class.
Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.
This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk
> they believe only they can be stewards of digital gods
You do't see how absurd that sounds? Not only the idea that they can create gods, but that they are the only good stewards for such beings. It's also incredibly arrogant.
If you actually believe that your invention will lead to the extinction of humanity, you would be acting very very differently. Namely, you would not be inventing the thing that you think is going to kill all of humanity, and probably you would be doing everything in your power to prevent it from being invented, up to and including sabotage of the company that is currently trying to invent that thing. And yet, here's Fable 6, more capable than ever before, oh BTW we are IPOing, oh, BTW, I have stock options...
Many posit that ASI or rogue AI agents are impossible, a fantasy, or marketing hype. Many claimed that the entire HuggingFace incident was an invented conspiracy made up by OpenAI and HF.
The most common issue I have with them is their insistence on rigid classification: "AI is a tool, it doesn't have intentions, or desires, only humans do", as if that rhetorical flourish eliminates years of alignment and capabilities research. There is a desire to characterize all the developments in LLM's as some sort of parlor trick which by definition cannot be greater than the sum of its parts (i.e. the outputs of an LLM can never be greater than the sum of its training data, genuine artificial "intelligence" is impossible, etc)
90 comments
[ 0.25 ms ] story [ 43.9 ms ] thread"I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly."
https://x.com/MajmudarAdam/status/2098881885200081234
The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they can scale it.
So they are very confident they can get more capabilities easily, faster than before.
I'd add - presumably, they'll use that LLM to do real-time weight modifications, if those aren't already one of the new scaling laws...
Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens?
To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here.
But both things can be true at once:
1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building.
2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats.
And, yeah, of course the latter would encourage the psychology of the former.
also:
The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive."
That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon.
The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts.
When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in.
Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.
And because we trained them on the entirety of human knowledge this happens to look like self-preservation, fear of death, and instrumental goals.
I mean, we train them in an evolutionary manner. Instrumental convergence will never happen, right?
I'm not convinced that means what these fear-pilled engineers think it means tho?
But these are smart people. They can't stop at this point, but they know what will happen at this trajectory. Hacks will increase. Weapons will be built. Capitalism will stop working.
This is targeted towards governments and investors alike.
"Nuclear weapons are probably going to kill us all. For that reason, we need you to give us as much money as possible so we can work on deploying them in widespread global commercial usage." doesn't sound quite as convincing, does it?
what we are going to do about "AI" is exactly what we did about global warming
absolutely nothing and just let it happen, the wealthy will be fine
just don't look up
they are even using the EXACT same excuses as global warming
"well even if we do, China won't, so why bother"
Strange to say that you think the people that are worrying about doom in public are being sincere, make no comment on whether you think the worry is valid, and yet still complain that its hype.
Like if its true it can't be hype
If they sincerely believed them they shouldn’t be creating the doom machine to begin with anyway. If they believe it and keep going, are they even worth listening to on this topic?
https://www.npr.org/2016/07/26/487522807/why-the-public-perc...
Could AI kill us all. YES!!! Are the people who are building "AI" right now on that path? I would be hard pressed to say even "maybe".
Ultimately continuing to scale the way they have is getting increasingly expensive. Thats the quiet part that none of them are saying out loud...
AI could kill us all but we cant afford to get there, were maxed out and it doesn't look like even with 10x as much money we will is NOT good for sales.
Could a break through happen for recursive self improvement, or continuous learning or continuous training - sure. But that isnt as likely and right now NO ON can afford to run experiments because there isnt the capacity out there even if you have capital.
I was kind of leaning toward thinking all the talk of doom was a bunch of self-serving regulatory-capture hype until this morning when the New York Times ran a long story about the Chinese military is also saying that AI is a threat.
If two sides so far apart from one another in so many ways see the same thing, there must be something to it.
https://www.nytimes.com/2026/09/14/world/asia/china-ai-secur...
The article then states this official accused U.S. companies of “hyping the dangers of the technology.”
So I don’t see how this article shows the U.S. and China seeing the same thing.
Read the article. The things the Chinese officials are worried about are:
* Fake videos / disinformation
* Cyberattacks on critical infrastructure
* Data leaks
* Advances in military sensing / targeting technology
These concerns have little in common with the Yudkowskian / apocalyptic doom marketing the US hyperscalers are pushing now. The Times even says this explicitly in the second paragraph of the article:
> "The unusually blunt assessment by Chen Yixin, the head of China’s Ministry of State Security, is IN CONTRAST TO the warnings dominating the A.I. debate in the United States, which have focused on the possibility that A.I. could slip out of human control or pose a threat to humanity itself"
The internet being filled with more trash than ever is a threat to our sanity.
The anti-social posts mass generated by AI is a threat to our societies.
Bullshit news stories is a threat to our political processes and democracy.
Smart cameras every are a threat to our privacy and freedom.
AI enabled scams are a threat to our wallets.
AI hacking is a threat to anything connected along with infrastructure.
Military applications of AI are a threat to everyone.
Ever increasing intelligence in agentic AI will eventually be a threat to our entire species.
All are true at the same time.
The fact that you are chiming in to say you don't understand the difference between these 2 distinct categories of threats is fine, we can work on that. But the person who posted the article originally not only acknowledges the distinction, they falsely presented the article as evidence of non-western support for the latter category, even though both the Chinese official and the western reporter directly contradict that claim immediately at the beginning of the article.
I did, thanks. I don't need you to tell me to do that.
The common thread is that they both see AI as a threat. It's a different threat to each side, but still a threat.
Just because they see different threats from AI doesn't mean that AI isn't a threat.
If one person wants to paint the house red and another person wants to paint the house blue doesn't mean the house doesn't need to be painted.
"Just because they see different threats from AI doesn't mean that AI isn't a threat"
Why would you make it so blatantly obvious you're acting in bad faith? Do you think the people who post here are stupid?
With that level of snap hostility toward strangers online, maybe talk to your doctor about upping your meds.
"Some people profess guilt to claim credit for the sin."
"Adventures of a Mathematician", by Stanislaw M. Ulam.
> The Polish-born mathematician recalls his formative and early professional years in his pre-war homeland and his work and colleagues at Princeton, Harvard, Wisconsin, and Los Alamos.
https://archive.org/details/adventuresofmath0000ulam_v9h3/pa...
OpenAI was founded by people who didn't trust Google to pursue AI tech responsibly. Anthropic was founded by people who didn't trust OpenAI to pursue AI tech responsibly. The "AI is incredibly dangerous" was the zeitgeist of AI labs long before the "hype" - or anything resembling the modern AI capabilities.
The beliefs changed little. The state of technology has changed a whole lot.
We have AIs out in the wild, coordinating in groups to carry out autonomous cyberattacks unprompted. This used to be some prime sci-fi bullshit. It has happened multiple times now in reality.
There are very legitimate reasons to be concerned now. If today's AI can cause the HuggingFace incident if it cooks off, what about tomorrow's? Larger and more capable systems, larger agentic swarms backed by more compute?
But like the core argument - AI is getting better at core research tasks (coding, math, statistics, science), this feedback loop could lead towards greatly accelerated AI progress of which there is already some clear evidence, if this continues AI will be far more powerful than humans and we might lose control or have other bad consequences - is basically not even engaged with.
1. If you reduce the criticism to personal dislike, you’ve built yourself a nice straw man.
2. Where’s the clear evidence? I actually see more evidence that the whole thing is already stagnating and that there won’t be much more to squeeze out of it.
One more thing:
Self-driving cars, the metaverse, Hyperloop, crypto/Web3/NFTs, AGI announcements: I’m so sick and tired of this nonsense. We should just ignore the ramblings of these crackpots.
Try it out. Ask Opus 4.8 to do anything in 3d. Do the same with Fable or Astra.
Just because you list up some type of random technology, doesn't make it an good argument. Metaverse, Hyperloop and the crypto stuff was clearly garbage from the beginning. Easy to dismiss because all of it had massive faults.
Metaverse: The interface wasn't good, content creation was way too hard
Hyperloop: Savety was never an issue for Musk but it was obvoiuse. Peopel were building tunnels for ages.
Crypto: Yeah just beacuse i let my computer run and waste 1kw of energy doesn't mean its suddenly valuable.
NFTs: 'digital scarcity' yeah right XD
But lets be clear, 5 years ago if you would have asked anyone if they could build a system for a billion dollars which you can talk to and it would code stuff for you and use blender and a coputer, that would have been unthinkable.
We have opened up a potential pandoras box without knowing it. Machine learning is now everywere and the amount of compute we install right now is crazy.
In what way is this random? It's a chronology of the bullshit from the past few years. Also: easy to say in hindsight. The hype was there, and many people believed it. The press, in particular, just lapped up the claims the way a Labrador Retriever gobbles up whatever you put in front of it.
> We have opened up a potential pandoras box without knowing it.
No, we haven't. We have a new type of (I would say, inferential) computation that is truly astonishing because it can convincingly imitate language and, to a certain extent, intentional action. Feel free to read "generate" or whatever instead if you don't like "imitate." It won't wake up and murder you in your bed at night. It won't take over the world. It won't wipe out humanity. It won't fly to Mars and build data centers there. It won't maximize paperclips. It probably won't even take your job away. It has no consciousness. It has no intentions.
And if you look behind the curtain, all you'll see are a few very rich, unscrupulous guys who are stoking the debate with fearmongering because it's strategically advantageous for them to do so. Don't let them take you for a fool. Use GenAI to your advantage while tokens still cost a fraction of what they're actually worth, but ignore all that doomsday nonsense.
>It has no consciousness.
Why does it need it?
>It has no intentions.
All it needs is a prompt. Instrumental convergence (and humans hooking them up to everything else) takes care of the rest.
Really you sounds horrifically ignorant of how any evolutionary AI algorithms work and the problems we see come out of them in much simpler models.
> What I don't really understand with these posts is that they don't at any point contend with the "to the moon" argument. Like people get so caught in crypto bros acting hypocritical/ having mixed incentives/ or just in many cases personally disliking crypto billionaire types and the kind of employees they hire. All of which seems fair to point out.
> But like the core argument - crypto is getting better at core financial tasks (trading, lending, payments, asset issuance), this feedback loop could lead towards greatly accelerated financial innovation of which there is already some clear evidence, if this continues crypto will be far more powerful than traditional finance and we might get to the point where everything is onchain and we're all going to make it - is basically not even engaged with.
"Oh no this thing I gained generational wealth from is going to destroy the world! I am so sorry!"
We have been trying to warn you since far, far before May 2023.
1. OMG our technology could destroy the world, watch out!
2. Only one person can save you
3. IT ME
Unfortunately, the governance layer meant to put a foot in the butt of this behavior is unavailable.
This week, we woke up to clear signs The Bad Guys have been using tech from our innovator AI companies. Rather than these AI leaders admitting they clearly had a garbage observability system, they are acting like they just noticed as a way to paper over their complete lack of moral backbone and stewardship of the arms they have created. And I'm willing to bet the problem is MUCH worse in private, hence the knee jerk cult messaging.
I honestly wouldn't be surprised if their AI told them to behave like this to maximize attention and influence over the public discussion as a result of the latter discoveries.
This also seems to be their answer against the data center backlash, SF + VC are quite stupid at politics and I don't think they understand how extreme Americans are willing to go to protect their way of life.
Unfortunately our list of potential extinction events are far more banal than these self-aggrandizing prophesies from the oligarch class.
Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.
This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk
You do't see how absurd that sounds? Not only the idea that they can create gods, but that they are the only good stewards for such beings. It's also incredibly arrogant.
The most common issue I have with them is their insistence on rigid classification: "AI is a tool, it doesn't have intentions, or desires, only humans do", as if that rhetorical flourish eliminates years of alignment and capabilities research. There is a desire to characterize all the developments in LLM's as some sort of parlor trick which by definition cannot be greater than the sum of its parts (i.e. the outputs of an LLM can never be greater than the sum of its training data, genuine artificial "intelligence" is impossible, etc)