Status pages are basically useless if they’re public facing.
Either they automatically update based on automatic tests (like some of the Internet backbone health tests) or they’re manually updated.
If they’re automatic, they’re almost always internal and not public. If they’re manual, they’re almost always delayed and not updated until after the outage is posted to HN anyway.
I prefer fully automated tests publicly revealed because the main thing I want to know (as a customer) is should I keep trying to fix my end or give up because GitHub exploded again.
It’s most annoying when you have something like recently - known maintenance work on my upstream home fiber connection that was resulting in service degradation (but not complete loss, my fiber line was back to DSL or dialup). The chat lady could see that my area was affected, but the issue lookup system couldn’t.
If the issue lookup had told me there as an issue I’d’ve gone on my merry way.
I even checked a few more times until it was resolved; the issue never appeared in the issue lookup system.
This was much much much easier when websites used to explode with tracebacks and other detailed error messages, now you just get a "whoopsie doopsie we did a fuckywucky" and you can't really tell what's going on.
you can't operate at any scale at all without mechanisms in place to know perfectly well whether an issue is impacting a single customer or if your world is on fire
You'd like to think so, but surprisingly large number of "large scale" things operate on the "everything is fine" until too many people complain about the fire.
Quite often you see automated tests that check how well your cache/in memory data are working. But when some other customer that isn't in the hot path tries to access their request times out. I've seen a lot of people making automated checking systems fail at things like this.
The other problem with status pages is depending on what happened it may not be possible to update the status page anyway. You really need a third party to have a useful status page.
Currently there is a banner on the AT&T outage page with this message:
>Service Alert: Some of our customers are experiencing wireless service interruptions this morning. We are working urgently to restore service to them. We will provide updates as they are available.
It would be nice if the FTC mandated this. It is exhausting when the status page is taken over by the marketing department (the infamous green check with the little "i").
I'll take a chance here and say this was a hack, possibly at the equipment-level. One major carrier having an outage? Possible. But three? On different networks?
Even if its not a hack I'd love to see the root cause on this one! Communications is critical infrastructure, so I'm gonna guess the government will demand a full report.
There is some shared infrastructure in the PSTN that all network operators use. This smells more like an SS7 outage to me than a hack, but we'll have to wait to find out.
Back in my circuit switched days, we lost half of our US long distance routes, because a farmer in Wyoming dig up an unmarked fiber link. Hence, backhoe fade.
When you lose your own network, your phone connects to a different one to be able to make emergency calls. One carrier knocking down other carriers is entirely plausible.
"A Verizon spokesman reached by CNBC said there was no issue with the Verizon network and their customers are only impacted if they try to reach out to the carrier experiencing the problem."
On the latest iPhones, SOS mode is the emergency fallback to satellite service. It's really meant to be used in situations where you're well outside of any sort of service area but you have a clear, unobstructed view of the sky.
Your iPhone will instruct you on where to point and help you track an emergency satellite that is manned by live humans who will take your emergency request and relay it to the proper people.
I’m pretty sure that “SOS only” can also mean the phone seeing networks it can’t register with but which it could make an emergency call on if required. This predates satellite SOS.
I'm wondering if it could be some kind of auth timeout. I've heard from a few people of one person's phone going out, then a bit later the other person's phone finally failing too.
Good thought, but switching to LTE only didn't work. Same result of ending up in SOS only. Cellular over wifi works perfectly fine though. Wish we could count on better post mortems from the phone companies, but I'm not holding my breath for it.
I wonder how SIM registration works? For example if it's like a token with an expiry. If some set of registration servers on the network couldn't renew then I could see behavior like this.
There was an earth directed X class, but not a big one. If that could affect SS7 I would expect it to also take out a chunk of the internet but I am not seeing that. [1] There are many cellular networks having issues but some could be and likely are resellers of others. [2] Probably more likely SS7 related as marcus0x62 mentioned. Another potentially SS7 related possibility? [3]
I know the other carriers are saying they aren't affected, but one look at DownDetector shows that nearly every carrier was affected, and all at the same time.
A user who has a working cell phone could contact a user who is on a network experiencing issues. Because the call fails, that user may decide their cell network is also faulty. Downdetector only works on user reports. Its basically useless for actual measurement because people are bad at troubleshooting.
That site counts mentions over social media, if you said 'T-Mobile/Verizon isn't having an outage' it'd still show up as outage activity on it. Plus people report issues calling AT&T customers.
I haven't had service since around 3am this morning, my internet (fixed wireless) was also down around that time but has since come back on. I had just assumed it was local tower maintenance until I woke up and see it's still out and it's effecting millions. But was surprised to see it barely mentioned/getting engagement here given the scale.
Saw my Comcast was down also (SW Chicago suburbs), about 3AM CST, but it was back before 4AM. I figured it was a neighborhood outage when rebooting the cable modem did nothing, and then it came back without any fanfare.
My T-Mobile phone hasn't had any problems, knock on wood.
I'm also in the southwest suburbs, and have Comcast. My internet went out for about ten minutes at around 9:20 CST. I disabled WIFI on my iPhone only to discover I was still offline, and had no signal. Not sure whether the Comcast outage was related, but it came back up very quickly, whereas I still have no cell service (AT&T). My wife does have service, even though we both have iPhone 15 Pros.
Reporting status externally is hard. I worked at a cloud provider and while we had very detailed metrics about how our systems were functioning, it was difficult to distill that information in a way that customers would understand if they were being impacted or not, and what the impact would be. Just reporting raw numbers wouldn’t give customers the context to understand what was actually going on.
How do you actually report, for example, that .003% of your customers are having a really bad day but the rest are just fine?
Aren't these exactly what p99 illustrates? Of course you'd have to have your use cases logged well enough to be able to aggregate the 99th percentile accurately to each customer flow
because you define thresholds for event classification.. the difference between 10024 customers having failure and 10100 customers having failure is not the question, right? when many hundreds of thousands of customers are failing at once, is that really very difficult to determine?
secondly, there are financial, management and information security pressures to NOT REPORT reality to the public. This happens VERY OFTEN in real business. In fact, that is why legal enforcement actions and real consequences are crucial versus Big Business.
In a cloud, massive outages are the rare events, and technically easy to report.
Small outages happen all the time, and are difficult to report accurately.
I think AWS has pivoted to trying to report status in each individual customer's support portal, so that they can give a dashboard that reports the state of the cloud from that customer's perspective. That way a rack down that only affects a few customers is only reported to those customers, and the dashboard doesn't have to always be red for everyone (or green for everyone, even those affected).
I wonder if this is a cyber incident. Curious if any telecom folks know what the most likely explanation for an event like this would be, and what telltale signs/symptoms might first indicate this was caused by something nefarious.
Kind of weird that there's been almost no coverage of this event this morning. CNBC has barely mentioned it. All 3 carriers having a major outage seems like it should be major news.
When I go to cnn, Washington Post, and NY Times all three have big stories about it prominently on their website, so it does seem like it’s being reported on fairly widely.
I wonder if this is a cyber incident. Curious if any telecom folks know what the most likely explanation for an event like this would be, and what telltale signs/symptoms might first indicate this was caused by something nefarious.
> However, the US Cybersecurity and Infrastructure Security Agency is “working closely with AT&T to understand the cause of the outage and its impacts, and stand[s] ready to offer any assistance needed,” Eric Goldstein, the agency’s executive assistant director for cybersecurity, said in a statement to CNN.[1]
from the same article above, it seems like it's a critical part of this.
> “Everybody’s incentives are aligned,” the former official said. “The FCC is going to want to know what caused it so that lessons can be learned. And if they find malfeasance or bad actions or, just poor quality of oversight of the network, they have the latitude to act.”
If AT&T gets to decide if they are at fault, they will, of course, never be at fault. So a third-party investigation makes a lot of sense.
I would also suspect that the FCC would not be as well versed in determining if there was a hack or even who did it, which is why I feel like CISA would need to get involved in the investigation.
like, you could commit a dumb BGP config and break lots of stuff. have done that in the past, actually...
but any time a national-tier ISP has a national-level outage, that warrants a look from multiple orgs. and given the number of threat actors like china, NK, iran, and russia, who are, and have, made aggressive efforts in this space -- and have strong reasons to do so now -- its not crazy for the US fed'gov to want to know a little more, and offer to help. but again, entirely possible it's unrelated.
I can confirm that San Antonio, Dallas, and Austin are all down. My dual SIM iPhone shows 0 bars for AT&T and Google Fi (T-Mobile) is fine. My wife's phone just shows SOS.
I'm just curious how we as a society manage the uptime of such a critical service? Do we have laws or enforcement in place to regulate / enforce how well basic cellular service (at least the emergency tier) must work in the US?
Just imagining the dropped emergency calls today, etc?
usually redundancy. but as these companies squeeze more and more out of gear, we risk large cascade failures that are hard to recover from. could also have been some kind of firmware upgrade. anecdotally, the 4G LTE phones seem to not be affected whereas the newer 5G ones seem to be. maybe it was a botched deployment. i'm sure ATT will give us a really well rounded RCA ;)
It can (FirstNet's lower 700mhz spectrum), but AT&T uses the same backbone and infrastructure and towers for large chunks of their and FirstNet's network.
(charitably) - FirstNet is mostly now just a type of billing plan and priority level, on AT&T's network.
FirstNet absolutely went down in my locality (western Ohio, near Dayton). I was on shift for after hours support of a public safety answering point (AKA a 911 center) and I received a call at 04:14 Eastern stating that all the law enforcement/fire mobile units on FirstNet disconnected from the VPN and that no FirstNet cell phones could make/receive calls or send/receive text messages.
how does a landline become relevant when one is not at home? right up to the start of work from home, I was spending very little time at home. a landline made no sense at that point.
Even a landline could have emergency calls disrupted if a line was cut somewhere. unless every device was behaves as a router with more than one signal in/out, there will always be a potential for loss of signal and interruption of service.
911 support is required of companies, it's monitored and fixed when not working.
But in this case, it's redundancy from the various providers - all providers must let you make a 911 call, no matter your phone or contract or provider.
So if AT&T's towers go down, your phone can still make 911 calls via Verizon or whomever else has a tower in your area, anything that can hear your phone will respond to a 911 call request.
You should be able to connect to a different network (e.g. Verizon) and make a 911 call even without a plan. You won't be authenticated to the network for data, but 911 is supposed to still work.
No no no, see, we have those. But we only use them to make sure that our for-profit prison complex stays massively profitable and the people in power retain that power.
Verizon and T-Mo both issued statements that they have no outages and the issue is just their customers being unable to call AT&T customers. Looks like most of the AT&T network in the US is down though.
I've been tethering with T-Mobile as my primary internet connection and that's been working just fine. Voice also works for me with both TMo and Google Fi.
A theory for the reported Verizon/T-Mobile issues is that when AT&T went offline, all of those phones went into SOS mode and tried to register on the remaining available networks (Verizon and T-Mobile) to allow 911 calls to be made. The surge in devices registering at once may have overloaded some parts of those networks.
You don't need to register to allow 911 calls. You 'register' (it's not a regular registration) at the moment you are placing the actual 911/112 call. At least that was in 2G/3G networks, doubt it changed.
There is always some amount of terminals without SIM or without working SIM, there is no need for them to bang every available network in the vicinity just in case there could be an emergency call.
Seeing "SOS" only on iPhone currently. I got worried something had gone wrong with auto-bill pay since I only noticed after I was driving.
It's interesting how naked I feel without access to the internet. I reach for it way more often than I would have ever guessed, something you only notice when it's not there. Last March my area saw large wind storms that knocked out power for almost a week (I'm not in a rural area). I can work around the loss of power but the cell tower(s) that service my area could not handle the load and/or the signal in my house was weak and I was unable to load anything. Not having internet was way worse than not having power and I ended up driving a few hours away to my parent's house instead of staying home.
It's even sadder - I used to be able to play computer games for hours offline, now I get about five minutes into even the ones WITH on offline mode, and I'm grabbing for a wiki or other reference. Ah, some of it is just getting old.
I think some of it is just not having oodles of free time to figure it out on your own. When I was young I would just keep trying things till I figured out the game, my time wasn't worth much or at least I didn't value it highly. Nowadays I don't want to spend 1-3 hours figuring something frustrating that's blocking my progress. The "rush" I get from solving it on my own does not make up for the time lost. Also I feel like games made today almost expect you will need the wiki/guide to figure out certain things. Or at least I often think "How the heck was I supposed to figure that out?" when reading the wiki for some aspect of a game I'm stuck on.
That latter part is certainly true, only a few games "offhand" even really try to work "wiki-free" (Factorio is perhaps the best here, but Minecraft is trying).
So.. it's an adult strategy for playing video games to extract a candy coated "win." We are all either overgrown children and always will be, or something has gone drastically wrong in the schools.
Lexington, KY. Not a massive city but the second largest in KY. I left after 2 days of no power and it didn't come back on for another 3-5 days more after that depending on where you lived.
Portland checking in. Those storms were gnarly and there was carnage all around us. Luckily we maintained power and internet. We have 11 month old twins and a three year old so 10 days without childcare or help was its own challenge.
Yes I've felt the same way. I feel like we have an instinctual need for social connection that we've filled with internet. Luckily, we do still have meat-space friends and family.
Yeah, that was a big reason I went to stay with my parents/family. I felt super isolated from my friends (local and remote) when I couldn't participate in group chats/communicate. Also I just kept picking up my phone to look up something or check on something only to re-remember I couldn't do anything. I had podcasts and audiobooks on my phone which helped but the isolation was a weird feeling I hadn't felt before. After I thought about it I realized it had probably been a decade or more since I had been completely without internet for more than a few minutes. It was odd...
I started driving across the US at 3am, didn't notice for the first few minutes until I tried pulling up the address in Apple Maps. Sure was strange following interstate signs for ~10 hours!
484 comments
[ 3.0 ms ] story [ 320 ms ] threadThey need to be accurate. At&t status claims everything is fine.
My wireless service is down. Down detector has tens of thousands of reports, so clearly everything is not fine.
Either they automatically update based on automatic tests (like some of the Internet backbone health tests) or they’re manually updated.
If they’re automatic, they’re almost always internal and not public. If they’re manual, they’re almost always delayed and not updated until after the outage is posted to HN anyway.
It’s most annoying when you have something like recently - known maintenance work on my upstream home fiber connection that was resulting in service degradation (but not complete loss, my fiber line was back to DSL or dialup). The chat lady could see that my area was affected, but the issue lookup system couldn’t.
If the issue lookup had told me there as an issue I’d’ve gone on my merry way.
I even checked a few more times until it was resolved; the issue never appeared in the issue lookup system.
Making this decision easy is a fight I fight for my customers every day. :)
Quite often you see automated tests that check how well your cache/in memory data are working. But when some other customer that isn't in the hot path tries to access their request times out. I've seen a lot of people making automated checking systems fail at things like this.
>Service Alert: Some of our customers are experiencing wireless service interruptions this morning. We are working urgently to restore service to them. We will provide updates as they are available.
https://www.att.com/outages/
It would be nice if the FTC mandated this. It is exhausting when the status page is taken over by the marketing department (the infamous green check with the little "i").
I'm not familiar, what are some examples?
Even if its not a hack I'd love to see the root cause on this one! Communications is critical infrastructure, so I'm gonna guess the government will demand a full report.
Back in my circuit switched days, we lost half of our US long distance routes, because a farmer in Wyoming dig up an unmarked fiber link. Hence, backhoe fade.
Sounds like only one had an outage
Seems like they are doing drone testing. Hmm, should we use drone network that will interfere with mobile phone network?
I wonder if the MVNOs that piggyback on AT&T are showing down also. If not, it’s some AT&T service authorization system that exploded.
Your iPhone will instruct you on where to point and help you track an emergency satellite that is manned by live humans who will take your emergency request and relay it to the proper people.
More specific info here: https://support.apple.com/en-us/104992
If there is no cell service then it's SOS with a little picture of a satellite next to it.
So the radio bands may play into it although I would think with latest iPhones, they can use any of the bands from AT&T although I could be wrong.
I suppose I can't speak to likelihood with a sample size of one.
[1] - https://www.thousandeyes.com/outages/
[2] - https://downdetector.com/
[3] - https://www.webwire.com/ViewPressRel.asp?aId=318230
Edit: I’m getting 1 bar when I usually get 4 in southern Ontario. But I see no broad reports of issues.
Could be a hack Could be a single point of failure Could be a config change that borked the system
Could be other things that make more sense we will have to wait for more info.
We did have Two Class X flares from sunspot 3590 but they did not result in a coronal mass ejection.
There was a CME from another sunspot not visible but it is not aimed at the earth.
https://www.swpc.noaa.gov/news/two-major-solar-flares-effect...
My T-Mobile phone hasn't had any problems, knock on wood.
How do you actually report, for example, that .003% of your customers are having a really bad day but the rest are just fine?
secondly, there are financial, management and information security pressures to NOT REPORT reality to the public. This happens VERY OFTEN in real business. In fact, that is why legal enforcement actions and real consequences are crucial versus Big Business.
Small outages happen all the time, and are difficult to report accurately.
I think AWS has pivoted to trying to report status in each individual customer's support portal, so that they can give a dashboard that reports the state of the cloud from that customer's perspective. That way a rack down that only affects a few customers is only reported to those customers, and the dashboard doesn't have to always be red for everyone (or green for everyone, even those affected).
Because those cloud provider internal tools are often wrong about who's impacted and who's not.
Me with a Pixel 7 Pro (dev preview android) on the same plan: not affected.
Strange.
[1] - [https://www.cnn.com/2024/02/22/tech/att-cell-service-outage/...
This isn't telling of anything, right? Wouldn't CISA be involved with anything that impacts Public Infrastructure at this level?
> “Everybody’s incentives are aligned,” the former official said. “The FCC is going to want to know what caused it so that lessons can be learned. And if they find malfeasance or bad actions or, just poor quality of oversight of the network, they have the latitude to act.”
If AT&T gets to decide if they are at fault, they will, of course, never be at fault. So a third-party investigation makes a lot of sense.
I would also suspect that the FCC would not be as well versed in determining if there was a hack or even who did it, which is why I feel like CISA would need to get involved in the investigation.
like, you could commit a dumb BGP config and break lots of stuff. have done that in the past, actually...
but any time a national-tier ISP has a national-level outage, that warrants a look from multiple orgs. and given the number of threat actors like china, NK, iran, and russia, who are, and have, made aggressive efforts in this space -- and have strong reasons to do so now -- its not crazy for the US fed'gov to want to know a little more, and offer to help. but again, entirely possible it's unrelated.
Just imagining the dropped emergency calls today, etc?
(Genuine question)
Knowing most megacorps, they'll blame some midlevel engineer doing a "bad" thing, when it was ordered from above.
(charitably) - FirstNet is mostly now just a type of billing plan and priority level, on AT&T's network.
CNN reported that AT&T confirmed it has not been impacted throughout this event.
https://www.cnn.com/business/live-news/att-outage-02-22-24/i...
https://www.ooma.com/blog/home-phone/landline-home-phone-is-...
Carriers do get fined for E911 downtime IIRC. It is taken very seriously, as a result.
People/companies that actually care already have a second line and are taking very good note of this.
But in this case, it's redundancy from the various providers - all providers must let you make a 911 call, no matter your phone or contract or provider.
So if AT&T's towers go down, your phone can still make 911 calls via Verizon or whomever else has a tower in your area, anything that can hear your phone will respond to a 911 call request.
No no no, see, we have those. But we only use them to make sure that our for-profit prison complex stays massively profitable and the people in power retain that power.
Do I believe that? No clue. I believe it more than people speculating the timing corresponds with the solar flare or nation state taking it out.
You don't need to register to allow 911 calls. You 'register' (it's not a regular registration) at the moment you are placing the actual 911/112 call. At least that was in 2G/3G networks, doubt it changed.
There is always some amount of terminals without SIM or without working SIM, there is no need for them to bang every available network in the vicinity just in case there could be an emergency call.
It's interesting how naked I feel without access to the internet. I reach for it way more often than I would have ever guessed, something you only notice when it's not there. Last March my area saw large wind storms that knocked out power for almost a week (I'm not in a rural area). I can work around the loss of power but the cell tower(s) that service my area could not handle the load and/or the signal in my house was weak and I was unable to load anything. Not having internet was way worse than not having power and I ended up driving a few hours away to my parent's house instead of staying home.
Now my computer is insanely more powerful but without an Internet connection it feels dead and useless.
This happened to us with the recent storms a month or two back, some places didn't have power restored for 2 weeks+
Strangely sending to that phone from an iPhone looks like it sent, but nothing was received
Calling from Android to iPhone gives 'your phone is not registered', iPhone to Android plays a 'call could not be completed'
iMessage between iPhones seems okay