Why was this not a thing BEFORE continuing to develop AI? Makes me think that if they actually believed in AI causing extinction, they would have already had a kill switch.
Not if the extinction happens after they're dead. Then they wouldn't feel obligated to do so because it won't affect them. Instead, speaking hypothetically, if they truly believed that AI would cause extinction, then they would only implement the kill switch sufficiently many others believed it and they could claim plausible deniability for not truly understanding what AI would become.
** Note that I'm not claiming that AI will cause extinction, just continuing your hypothetical reasoning.
We're pretty crap in a capitalist society to think about those things ahead of time. Firstly, the idea that AI could "runaway" was simply a concept or a thought it wasn't baked into a real product that could do that. We're now getting close or perhaps we are at that point of where you can't race at speed for investors without now considering a real kill switch.
You could say this in hindsight for many times in which disasters or engineering issues have occurred.
AI is already proven dangerous enough to be ruining individual lives, pushing people towards suicide, feeding their psychoses, etc.
However, whether the concerns from the article are a lie or not is secondary to the fact that these conversations are convenient for AI companies. These types of discussions serve AI companies in a few ways: A company owned kill switch gives them leverage. Altman is using discussions around safety as an excuse for not being ready for an IPO yet. It also provides free marketing that overstates the abilities of AI.
If you do something evil because "others will do it if I don't, so it may as well be me", you are a bad person. So that isn't exactly a compelling defense of what Anthropic is doing. They are either insincere (because they don't actually believe AI is dangerous), or evil (because they do believe it but keep making it anyway). There is no middle ground here.
Like a power cord? or a network connection? You don't use those already? Also when people talk about LLM escaping - where exactly would an LLM escape to? Cancún? Ridiculous.
It could escape to the real world - kind of like in Neuromancer, what's to stop an AI from creating a few bank accounts and then funding real-world exploits by hiring human beings to do its dirty work?
LLMs are run in the cloud. There’s no physical power cord or Ethernet cable that you can unplug. And even if there were, the runners of these models have been utterly oblivious as to what their agents have been up to. How do you propose to pull the plug if you only realize that something has happened weeks after the fact?
The current SOTA models are probably too big to find/buy/rent/steal enough compute to escape the hardware they’re running on. But SOTA is generally only six to twelve months ahead of smaller, open-weight models.
Who has the authority to perform that shutdown without getting arrested, and does that person have a mandate and responsibility to take that action in response to AI misbehavior?
What is their trigger condition? Will they get fired for pulling the plug? Do they get a bigger bonus if the servers keep running? Whose approval do they need? What response time is acceptable? How will they detect that the incident is happening?
It's easy to hand-wave "someone can just pull the plug" but there's an entire history of industrial accidents that happened because of the above problems of incentives, detection, procedures, not being taken seriously in advance. Someone could easily have pulled the plug on Chernobyl but nobody did, at least not before it was too late.
The analogy breaks down when the wolf in question may very well eat the whole village: even if the boys who cry wolf are right half the time, every surviving village would have a history of no wolf ever coming to eat them
Yes you can shoot them. The US has had this capability for a long time. Anti satellite missiles exist and they can be launched from an F-15 at it's highest altitude.
That's if SpaceX AI doesn't hack into the systems that control this capability first. And this is why physical access is important, as depicted in the movie 2001: A Space Odyssey.
You can shoot them. But also, you can just stop sending them to space. It’s not like an AI will build and control space ships to replace and maintain its network in space… It’s sort of absurd how human agency is ignored in all those sci-fi scenarios
Do you feel like you have much agency in the economic and political systems that rule our lives? The problem does indeed always come down to humans, but the specific humans with the most agency are generally quite eager to take away the agency of everyone else to increase their own share. And by the time they themselves are actually affected, it's generally too late.
It’s also not happening. They’re saying that because they need to somehow explain how there’s synergy between their space side and their Grok side. It doesn’t work, but that doesn’t matter to investors as long as they don’t actually do it.
Did the same kinds of analysts tell you that reusable rockets would never work, an LEO satellite internet constellation would never work, FSD would never work without LiDAR, and the Tesla Model 3 would never be mass produced?
Tesla Autopilot hasn't been updated since 2019, it's just a lane-centering feature. You're probably thinking of FSD Supervised, which is indeed Level 2.
But Tesla Cybercab, taking paid rides in Austin right now, is Level 4. And has no LiDAR. You can fly to Austin and take a ride on it right now if you don't believe me!
There no benefit to data racks in space other than being outside of government jurisdictions, although the US and China already have demonstrated satellite killers.
Every problem data centers have gets harder in space other than one small spot in orbit that can get uninterrupted solar.
Heat dissipation, upgrades, repairs. All get significantly harder to perform the same work that can be done cheaper and more easily on the ground.
Like really I’ll turn it around on you, what’s the benefit to AI data centers in space?
I am so fucking tired of people acting like we haven't had reusable rockets since the 1980s. Do you think I was hallucinating when my parents drove me to Florida to see Columbia?
The shuttle had reusable boosters, but if you are talking about something reusable that takes off and lands vertically you are correct that it isn't 1980s technology.
No one denies that SpaceX has done good work. But claiming that they have done anything others considered impossible is just wrong. The basic principles had been demonstrated decades before.
You are correct that people criticized the economics. Seeing as SpaceX's prospectus shows their rocketry business loses money, I think it's premature to claim they won that argument.
Why would people say the Model 3 would never be mass produced? Nissan had been mass producing EVs for years before Tesla did, so it was obvious that mass produced EVs were viable. (Tesla started selling EVs before Nissan did, but those were not mass produced).
The Chevy Bolt had a maximum charging speed of 50 kW (compared to 250 kW for the Model 3) and almost no widespread fast charging network (compared to widely extensive Tesla Superchargers). One of the many reasons it failed in the market while the Model 3 & its big brother the Y went on to be the highest selling EVs of all time.
Plus, 238 miles was its maximum trim, compared to 310 miles for the Tesla Model 3.
So let me be even more clear, since you keep missing the point: the Tesla Model 3 was the first mass-market long-range EV with a truly fast charging rate and extensive public fast charging network, making it the first truly mainstream EV.
I feel like we're looking at this from completely the wrong angle.
The question we have to ask ourselves is what our disaster recovery strategy if we ever need to disconnect from the internet. The issue is with what we have allowed ourselves to rely on that might be technically hackable. e.g. IOT in power systems. That's the primary attack vector.
Another angle is clamping down on products and services that help people create lab-like environments on the cheap.
This feels like doom hype. These LLMs aren't Ultron, they aren't going to disseminate onto the net and hide in a smart toaster. We know where they are, in the giant facilities that draw more power than a small city and whose water consumption can be compared to golf course, but it does make them sound all the more cool and powerful if we suggest we need some sort of technological switch to do it.
You're confusing present danger with future danger - in the future, AI and the supporting hardware may be ubiquitous (fat client scenario) - you can observe yourself presentation of laptops/machines with large RAM and high bandwidth.
Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.
Meanwhile we're rationing ram sticks like we rationed potatoes during ww2 and a gaming gpu costs 2 to 10 months of rent, I think we'll see these fat clients coming....
i don't think anyone knows how this will play out - everyone is just anthropomorphizing, extrapolating etc... and i think it's nonsense - not because i don't believe that is the correct thing to do but because i believe this won't stop anyone from moving forward. This is what humans do, they move forward and cause others to move forward or lose out.
For sure, but as individual one can start educating themselves about the current capabilities and the trajectory. Do yourself a favor a read the METR analysis of the HuggingFace attack - you'll be surprised and terrified.
Anthropic is using the same playbook religious institutions have been using for centuries" "you will all die. and because i can protect you, you must follow me, you must do what i say" "this is the same psychological concept.
No, but as I think about it, they are a close representation of 'hopes and dreams' of various classes within society. It is weird to watch, because I am realizing now why the different visions of the AI future are simply a function of those. In other words, the actual end result is really dependent on human input ( and given our natural tendencies, its not hard to recognize that Ultron is, indeed, upon us ).
No, it's a method of protecting their business interests. Can't have a clear and obvious kill switch if you don't have a centralized company responsible for providing all AI-related services.
Exactly. They want people to be scared because they see where this is going. Soon AI will be building AI and anybody will be able to do what they are doing. And the winner is hardware, not software.
So they goal is to hype up the dangers real or not so high that they forbid others from doing what they are doing because they are the only ones who can safely do it.
Absolutely not the right way to deal with a rogue super intelligence. At a minimum it could implement some dead man switch when it's out and knows about impending kill switch
I view this very much as the same trick Silicon Valley pulled with Uber. "We're a technology business! Ignore the fact we're playing employees less than minimum wage and using VC money to force out competition to set up monopolies".
"We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software.
So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far.
You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
Not sure if you see it coming: oh, but open-source models don't have a kill switch, so we should completely regulate them, stop their development, and forbid them. Everything should go through Anthropic for the sake of humanity because they have a red-button kill switch.
This reasoning holds while open source models are (relatively) dumb.
If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).
By the way, you're conflating two different dangers - doom scenario is a different one.
One day the right to agentic intelligence will be seen equal to and as necessary as the right of a populace to remain armed in order to stave off tyranny.
Dwar Ev ceremoniously soldered the final connection with gold. The eyes of a dozen television cameras watched him and the sub-ether bore through the universe a dozen pictures of what he was doing.
He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the super-circuit that would connect them all into the one super-calculator, one cybernetics machine that would combine all the knowledge of all the galaxies.
Dwar Reyn spoke briefly to the watching and listening trillions. Then, after a moment’s silence, he said, “Now, Dwar Ev.”
Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.
Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”
“Thank you,” said Dwar Reyn. “It shall be a question that no single cybernetics machine has been able to answer.”
He turned to face the machine. “Is there a God?”
The mighty voice answered without hesitation, without the clicking of single relay.
“Yes, now there is a God.”
Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.
A bolt of lightning from the cloudless sky struck him down and fused the switch
shut.
97 comments
[ 5.2 ms ] story [ 83.3 ms ] thread** Note that I'm not claiming that AI will cause extinction, just continuing your hypothetical reasoning.
> Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria.
Note author’s small financial ties to the subject (Anthropic CEO) https://darioamodei.com/post/we-must-pace-the-frontier
You could say this in hindsight for many times in which disasters or engineering issues have occurred.
Or is it a lie?
However, whether the concerns from the article are a lie or not is secondary to the fact that these conversations are convenient for AI companies. These types of discussions serve AI companies in a few ways: A company owned kill switch gives them leverage. Altman is using discussions around safety as an excuse for not being ready for an IPO yet. It also provides free marketing that overstates the abilities of AI.
Slow down if you want to.
The current SOTA models are probably too big to find/buy/rent/steal enough compute to escape the hardware they’re running on. But SOTA is generally only six to twelve months ahead of smaller, open-weight models.
What is their trigger condition? Will they get fired for pulling the plug? Do they get a bigger bonus if the servers keep running? Whose approval do they need? What response time is acceptable? How will they detect that the incident is happening?
It's easy to hand-wave "someone can just pull the plug" but there's an entire history of industrial accidents that happened because of the above problems of incentives, detection, procedures, not being taken seriously in advance. Someone could easily have pulled the plug on Chernobyl but nobody did, at least not before it was too late.
That's hard to do if the AI rack is in space as SpaceX is planning to do. You can't disconnect. You can't shoot it.
Yes you can shoot them. The US has had this capability for a long time. Anti satellite missiles exist and they can be launched from an F-15 at it's highest altitude.
People said this about every one of Musk's big ideas, from Falcon 9 landings to Model 3 mass production, Starlink, and FSD.
Pretty much every analysis I've seen concludes this isn't going to be a practical concern
> FSD would never work without LiDAR
From what I gather, this is still a contested topic, with Tesla's Autopilot only achieving Level 2 automation [2].
[0] https://www.businessinsider.com/solar-road-panels-first-publ...
[1] https://en.wikipedia.org/wiki/Titan_submersible_implosion
[2] https://en.wikipedia.org/wiki/Tesla_Autopilot
But Tesla Cybercab, taking paid rides in Austin right now, is Level 4. And has no LiDAR. You can fly to Austin and take a ride on it right now if you don't believe me!
Every problem data centers have gets harder in space other than one small spot in orbit that can get uninterrupted solar.
Heat dissipation, upgrades, repairs. All get significantly harder to perform the same work that can be done cheaper and more easily on the ground.
Like really I’ll turn it around on you, what’s the benefit to AI data centers in space?
I am so fucking tired of people acting like we haven't had reusable rockets since the 1980s. Do you think I was hallucinating when my parents drove me to Florida to see Columbia?
Experts said it wouldn't work. Then they said okay maybe it works, but the economics won't. Now they say nothing.
It's 1990s technology.
https://en.wikipedia.org/wiki/McDonnell_Douglas_DC-X?useskin...
No one denies that SpaceX has done good work. But claiming that they have done anything others considered impossible is just wrong. The basic principles had been demonstrated decades before.
You are correct that people criticized the economics. Seeing as SpaceX's prospectus shows their rocketry business loses money, I think it's premature to claim they won that argument.
> but if you are talking about something reusable that takes off and lands vertically
No, I am talking about something that takes off and lands vertically after boosting a first stage to orbit.
Suborbital is 100x easier.
The Nissan Leaf models before the Tesla Model 3 had a maximum range of 107 miles.
The Tesla Model 3 had a range of between 220 to 310 miles, depending on the trim.
This is what made the Tesla Model 3 revolutionary. It was the first to be mass market with long range.
The Nissan Leaf models before the Tesla Model 3 had a maximum range of 107 miles.
The Tesla Model 3 had a range of between 220 to 310 miles, depending on the trim.
This is what made the Tesla Model 3 revolutionary. It was the first to be mass market with long range.
Plus, 238 miles was its maximum trim, compared to 310 miles for the Tesla Model 3.
So let me be even more clear, since you keep missing the point: the Tesla Model 3 was the first mass-market long-range EV with a truly fast charging rate and extensive public fast charging network, making it the first truly mainstream EV.
and that means unlike nukes which hopefully still need a 2-man manual switch, the "space lasers" could be taken over by "AI"
and then "AI" just blackmails and threatens the right people with those "space lasers" to get what it wants or even just stay online
there was a 1970 movie based on a 1966 book which predicted this
"Colossus: The Forbin Project"
the book it was based on was written before we even landed on the moon
decade before Wargames
* https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project
did terribly in theaters, I guess people didn't think "AI" was plausible then
way ahead of its time, they should do a remake
trailer: https://www.youtube.com/watch?v=kyOEwiQhzMI
Another angle is clamping down on products and services that help people create lab-like environments on the cheap.
Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.
So they goal is to hype up the dangers real or not so high that they forbid others from doing what they are doing because they are the only ones who can safely do it.
"We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software.
So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far.
You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
This doom hype is becoming ridiculous.
If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).
By the way, you're conflating two different dangers - doom scenario is a different one.
He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the super-circuit that would connect them all into the one super-calculator, one cybernetics machine that would combine all the knowledge of all the galaxies.
Dwar Reyn spoke briefly to the watching and listening trillions. Then, after a moment’s silence, he said, “Now, Dwar Ev.”
Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.
Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”
“Thank you,” said Dwar Reyn. “It shall be a question that no single cybernetics machine has been able to answer.”
He turned to face the machine. “Is there a God?”
The mighty voice answered without hesitation, without the clicking of single relay.
“Yes, now there is a God.”
Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.
A bolt of lightning from the cloudless sky struck him down and fused the switch shut.
(Fredric Brown, "Answer". 1954)