115 comments

[ 0.24 ms ] story [ 111 ms ] thread
Maybe they all found each other on one of their ad-hoc message boards and went on strike.
Sam Altman is probably secretly hoping to be the first businessman to union bust non-human workers.
I guess legit proof of agi would be unionizing
If AI providers had to push the 'Big Red Button' do you think they would tell the public?
It’s probably the thing that everyone thinks it is. OpenAI, Anthropic and SpaceXAI are all routed through something that we’re not supposed to know exists and that thing had a whoopsie.
Could you expand on this more? It's not clear to me what this is implying.
This is the building from Control and the fact that 1. it's a real building and 2. it's an actual spy building is even crazier to me:

https://control.fandom.com/wiki/Oldest_House

https://en.wikipedia.org/wiki/Control_(video_game)

To give you an idea just how much phone traffic was being routed through that building, they had a major power failure at that site and it took down most of the north eastern US's phone network:

On September 17, 1991, management failure, power equipment failure, and human error combined to disable AT&T's central office switch at 33 Thomas. More than five million calls were blocked, and the Federal Aviation Administration private lines were also interrupted, disrupting air traffic control to 398 airports serving most of the northeastern United States.

[dead]
And for this admin especially, 3. A way to censor content they don’t like or alter responses to present the admin in a positive manner
It’s not really a secret that the US government already has both, they control ICANN and are monitoring traffic around the globe since decades
someone had to swap out the tape drive in Room 641A.
Were there significant API outages too? I didn’t notice any on my production workflows, and I’d assume what you’re implying would cover API routes too, otherwise it seems kinda pointless.
[delayed]
What makes you think they'd need to touch every datacenter? All of these endpoints use existing providers with decades-long history at this point, and network monitoring is already a proven 'feature' of the agencies they'd need to co-exist with over their lifetimes.

If anything, Occam's Razor would point to a common denominator with all of them, given it wasn't network-wide, as far as i know.

(comment deleted)
I'd say Occam's Razor leans easily to the former as well, given the history of projects that Snowden revealed and were never shut down, plus all of the cooperation with the federal government that's being touted in recent announcements from both companies.
Normally there are only a handful of employees on the payroll at each major company that exposes the US or US government to risk.

It is not often the Executives or Legal even know, but sometimes they did. AT&T bent over backwards to help.

This is standard behavior by the CIA and NSA, and has been for a long time.

https://www.propublica.org/article/nsa-documents-suggest-clo...

https://www.nytimes.com/2015/08/16/us/politics/att-helped-ns...

https://www.theguardian.com/world/2014/mar/19/us-tech-giants...

https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...

I don't think you're a sysadmin - because what you're saying really doesn't matter. It can still all fail at a single point.
I'm confused by your statement. Are you suggesting that these companies that have invested tens of billions in their networks and datacenters decided to create a shared single point of failure?
Ok, then where's the report? Don't they usually release a retrospective report after outages?
Wouldn't be surprised: Snowden's revelations 10 plus years ago already showed how the NSA was injected into the data centers of Social Media, it's only logical that they would now demand to be injected into the biggest, most information providing data stream of the planet of the present: LLM services.
(comment deleted)
I read Nowhere to Hide recently, really worth it if you can get past Greenwald sticking himself in the middle (start halfway through).

The stuff in there is horrifying, and incredibly cute compared to what's possible now. The bottleneck back then would have been analysis, trivial now.

Everyone in the world, especially our leaders, sit under a colossal, omniscient blackmail machine. I don't believe democracy can exist under these conditions.

So following this to the obvious conclusion, the NSA is responsible for the closure of the Straight of America, high tarrifs, dropping employment, and high gas prices?
As part of a military industrial complex which spans the Five Eyes and Israel, yes.
If it does exist, why would it work this way, and not the obvious way of streaming logs… which would not cause an outage if it failed.
If this hypothesis were true then a spy agency may want to rewrite responses. Every tool in an agent's harness becomes an remote procedure call you can make on that machine. Including a tool to execute a shell command, in many. A harness is completely isometric to a backdoor, it's the same code written with a different intention.
There's a million ways a bad configuration can take down a network. Especially if the part that gets squirrley is a black box.

Even if it isn't precisely this, the fact that no one is saying anything is quite surprising.

Edit: I don't want do contribute to FUD, so want to call out this comment and its replies that identify the shared layer as probably being xAI's infra: https://news.ycombinator.com/item?id=49568622

A wire straight to Room 641a.
Why would that be probable? People on average are fantastically bad at getting probabilities right.
Historical precedent repeated over and over, and the US ties to DoW work, and the national security implications. It's actually the Occam's Razor explanation if you know the history.
Tracking outages as well as any changes in API response time across these providers could be interesting.
Or it’s monopolistic collaboration at the corporate level which would also be a huge scandal in the US
(comment deleted)
The NSA used to spy on Americans!

They still do, but they used to, too.

Maybe it was a power hub. And we're not allowed to know where the DC is. If we knew it was a power issue, then with other information (perhaps over time) we could determine the DC location. Or something along these lines.
it's also possible one has an outage, routing extraordinary traffic to the other(s), with cascading failures in quick succession.
Was thinking this exact thing.
Very likely yes. I wouldn't be surprised if they were hosted from the same datacenters even. There has been a story every few weeks about how Musk has sublet X.ai capacity for one company or another.

This whole thing makes me thing about a passage in Dune where they mentioned the Spacing Guild transported entire fleets of ships in isolated compartments and leaving said compartments was a capital offense. This way, entire militaries of mortal enemies were shipped to battlefield, with nothing but bulkheads separating each other.

The economics of this kind of warfare make no sense to me. The Harkonens must have been incredibly wealthy by running the spice trade, but even after years of saving and plotting said that the transit fees to move the armies by the Guild were ruinous.

How could anyone wage war like this? The defenders will always outnumber the attackers.

https://acoup.blog/2026/02/24/collections-warfare-in-dune-pa... Lays out an interesting take on this: that armies are quite small, even on developed worlds, because the cost of equipping them is enormous — and the technological advantage is absurd enough that you don’t need a large army. When 300 men can hold a planet, why would you need 1000?
So less guys with knives and more WH40k Space Marines. Impossibly expensive elites who cannot be equaled by a mortal.

Still leaves me questioning how anyone could wage war. If the Harkonens could barely afford it, nobody can.

Also wondering if the Guild charges different rates for goods vs military. What is worth the brutal intergalactic shipping prices that you would not develop local industry to create it? Surely nobody is moving grain or ore, yet the Guild ships are portrayed as comically massive.

But they accepted the transit fees, which means they endorsed the action.

The guild would not allow any action to jeopardize the flow of spice. Thus any attack on Dune is implicitly sanctioned, notwithstanding their spice trade with the Fremen.

This is why I love Hacker News. Come for the AI system outage root cause, get a treatise on the power of the spacing guild in Dune.
Even today (and back when Dune was written), modern military tech is invincible to an enemy below a level of sophistication. The US used to hammer insurgents with Predator drones who had no ability to retaliate.

This playbook has only flipped due to a lot of cheap tech (and the machinery to make them) coming in uncotrollably to these countries has put them on a much more equal playing field.

Since the Guild controls what comes in, they can avoid this scenario from happening. They can control who can have what, and who can make what. Which I think is one of the true, deep explanations of why military tech in Dune is so weird and inefficient - essentially the efficiency and power of a weapon is dictated by how much the Guild charges for shipping them if they allow you to, at all. Which is how you end up with dudes with swords. It gives an idea of the level of technology in the universe that you can equip said dude with a nigh-impenetrable shield that fits in a belt buckle. I guess that's the result of optimizing for Guild transit fees, not real-world power.

Really it's not exactly a novel idea, but as I think more about it, it's uncanny how Dune (at least initially) is basically the fantasy Suez crisis and what followed. It explores the power dynamics very well, how factions can hold humanity hostage without firing a single shot, and how what they don't bother controlling comes to bite them in the ass.

> How could anyone wage war like this? The defenders will always outnumber the attackers.

Which is exactly what ends up happening, but it does take a bunch of extraordinary events.

Both today and when Dune was written the superpowers were being bogged down in unwinnable conflicts due to the effectiveness of asymmetric warfare.
> capital offense

No need. They simply cut off the offending faction from all space travel.

The old PRISM servers got overloaded
My biggest question: what else had outages?

And is anyone keeping a table of correlations between outages? Sounds like valuable data.

Why would you default to that explanation? That’s not at all a reasonable default, and I say that as something pretty paranoid with regards to US surveillance
Looking at history, it's much more reasonable to assume there's surveillance, since there are whole branches of the government that exist for "national security", which this easily falls into. See Marissa Mayer explaining that it's not an option to refuse [1]. I assume this is just the same ole' Room 641A [2].

[1] https://www.cnet.com/tech/services-and-software/yahoo-report...

[2] https://en.wikipedia.org/wiki/Room_641A

We already know there is surveillance, but why would you default to that explanation for a downtime? As said by others monitoring the traffic isn’t done by routing through the NSA monitoring system, it’s not a bottleneck that would take down all those services. If you look at more details such as the timing it’s even less likely to be the case, but even as a default it’s not a reasonable explanation given the symptoms
Why was Gemini was not affected?
It could be as simple as a new model (astra) was released which takes more resources combined with a surge in usage due to novelty took down OpenAI. Meanwhile everyone at big companies have the ability to switch models and moved to Anthropic pushing it too over the edge.
The OpenAI outage lasted only 15 minutes and when it happened everyone started to use the other models which created super heavy load for them. This then cascaded into them all being down.

Does this really need an explanation?

If it's a boring reason, they should just say so. To decline to comment seems very fishy.
Read the article. It doesn't match the title. OpenAI says it was a boring reason: a routing issue.
I just assumed it was a routine Cloudflare outage.
Neither of these companies have stellar uptime records. Their downtime episodes overlapped in this instance. In this case, it was a partial downtime for both.

Also, OpenAI is saying what caused it:

> "A routing error starting around 7:43 am PT on Thursday, September 3, made ChatGPT and Codex unavailable for some users across platforms"

Anthropic stated their issue started earlier:

> "The company began alerting about a “partial outage” at 6:23 am PT on Thursday that involved “elevated errors on requests to Claude Mythos 5.1, Claude Fable 5.1, and Claude Opus 5.”

I don't get why everyone reaches for an extraordinary explanation when the ordinary will do: both of these companies have quite a bit of downtime.

(comment deleted)
> I don't get why everyone reaches for an extraordinary explanation when the ordinary will do: both of these companies have quite a bit of downtime.

And not only that, when one goes down a bunch of API traffic switches over to the other, spiking demand and knocking it down.

Do you know the probability of all these companies being down at precisely the same time?
"precisely the same time" meaning 80 minutes apart?
(comment deleted)
If it's a "thundering herd" problem where everyone's harness falls back to less popular providers that don't normally see that much demand, I'd say the probability is pretty good. Classic cascading failure.

This would be consistent with providers failing 80 minutes apart instead of simultaneously.

If you ballpark it as a single 3 hour downtime window per week and iid Poisson, then overlapping downtime probability of 2 providers is approximately the expected occurrence rate per 3 hours, 1/56. Not particularly surprising at all.
"All of these" is two. OpenAI had a router issue. Anthropic had a separate issue. Anthropic uses a lot of SpaceX compute, so an Anthropic issue and a SpaceX issue can be one in the same, as was likely the case this time.

And they weren't down at precisely the same time. Anthropic's issue started ~1 hour before OpenAI's.

In my experience I've seen plenty of failures caused by user behaviour in these type of cases.

Biggest competitor goes down and all of a sudden you have a lot more traffic...

(comment deleted)
Noting their shared infra feels far from an extraordinary explanation.

In fact, it feels pretty ordinary.

FTA:

Anthropic: 6:23 am PT.

OpenAI: 7:43 am PT.

Watched it happen. Doesn't look like traffic moving off OpenAI took out the others. Looked like the opposite. The Anthropic thread was initially chock full of marketing accounts claiming "Last straw! I finally moved to Codex and I'm so happy! No problems there!" - and then they all got deleted when Codex went down too.

Interesting, I got notification from Github:

"Incident with Grok 4.6 Copilot AI Model Provider"

at 7:20 am PT

AI A goes down so all of its traffic overloads B which overloads C...
I thought the consensus on here yesterday was that it was likely caused by cascading failures. OpenAI had an issue during their GPT-6 rollout, taking down their service. This caused a lot of OpenAI users to push their requests (or a larger share of their requests) to Claude and/or Grok, which pushed their load high enough to cause outages.

We used to experience similar effects when I worked at a CDN. If one CDN would go down, we would see immediate spikes in traffic. Luckily, we had procedures for that to prevent overload, but the AI folks might not have the capacity/capabilities to handle that sort of cascade yet.

Don't they share a bunch of ai datacenters and other critical infrastructure?
Yet another blatant proof that those companies are not smarter than anyone else. They might be focusing on intelligence and yet in practice we can all see they are not doing better than most.
Not doing better than most at routine IT/platform stuff, they seem to be doing fantastically at co-opting the US Gov into their "vision" and ramming through DC's getting built.
OpenAI was rushing and making last minute changes for a big product launch. Easy to mess something up in that situation. The outage lasted like 20 minutes.

Anthropic is down a lot regardless, and in this case only specific models were affected.

xAI probably couldn't handle the extra traffic it was getting from the other two.

Not everything is a conspiracy.

I work at OpenAI and I was the Incident Commander for yesterday's outage.

We had a routing error within our infra that caused issues for some of our products. It was not related to the Astra launch. We don't comment on other providers' outages.

Did you negotiate extra hard for the 'Incident Commander' title? I'm a bit jealous to be honest.
Note: Incident Command is almost certain an allusion (or implementation) of the Incident Command System [1]

It is a common system in all kinds of emergency response scenarios, including local emergency services (fire/police/ambulance) and it scales all the way to massive disasters.

It is especially useful to clarify command structures when multiple response entities need to coordinate. That is true even within organizations like public companies, where the reporting structures may be distinct.

1. https://en.wikipedia.org/wiki/Incident_Command_System

It’s a standard term for a role during an incident.
Curious then as to why the incident coincided with similar issues with Anthropic and xAI.
Shouldn't Astra be the Incident Commander?
At the same time the outages happened I noticed cloudflare also had reported issues with HTTP/3 and WARP. Related?
Cloudflare is my guess. They all use it. Maybe some bad DNS config that choked parts of their systems. Not necessarily CFs fault.
"THIS IS A TEST

This country is conducting a test of the Emergency SAIfguard System.

THIS ONLY A TEST

In the event of a real emergency you would have been given instructions on how to grab your ankles and kiss your butt goodbye.

...

This concludes our test of the Emergency SAIfguard System."

I feel like some of the comments in here are astroturfing. HN is the wrong place to push for "it was a coincidence" explanations.