Today I’ve learned that vast swathes of the emergency call network depend on just one ISP. I wonder if the old, analogue and copper-only network wasn’t more resilient.
In Kitsap County (just across the water from Seattle) 911 traffic is delivered by Centurylink via SIP over a Wave Cable connection (think Docsis 3.0, not fiber). Why the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink.
Why the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink.
I've consulted with a few rural counties (I don't know a thing about the PNW, is Kitsap a rural or suburban county?) who got their circuits leased to them through a local MSP/IT Shop who had all the best intentions, but overlooked any sort of failover and you end up with an answering center taking calls over a single adtran going direct out to a CLEC.
Just wondering if this is less county refusing to do something, and more like the county implementing services sold to them that probably could have benefited from better planning and integration.
Probably the latter. How many rural county boards have people who actually understand any of this stuff?
It seems more than likely that Centurylink Sales/Marketing convinced a few counties to use them and then sold itself to other counties as "the" provider for such services.
How many rural county boards have people who actually understand any of this stuff?
Not enough, I would wager. It's something I spend a lot of time thinking about, living in one of the largest cities in the country now, with parents who live comfortably in the rural south-but have complained about the internet a couple of times.
To their credit, they'll lose connectivity for hours and not even mind, since mom will just turn on the radio and listen to the local PBS station and tend to her garden while dad goes out to his woodshop.
Others in their community probably aren't as content without connectivity, for such long periods as they are.
In NW WA you might be surprised. There's a long history of utility self sufficiency with the San Juan islands, and the area and neighboring counties has a lot of dark fiber available for lease from smaller independent providers. Many municipalities and community organizations light up their own WAN links.
There was a always a tension when it comes to outsourcing and considering core competencies in the orgs I saw. Can imagine less appetite for creative solutions when critical things like 911 are involved.
And what amazes me is that the ISP was content to be a single point of failure for emergency services. I'm no expert, but I'd think a primary design criterion for emergency systems is that they keep working in emergencies.
My friend and I were just talking about how they should put a bit of oil/natural gas at the top of all those little 911 repeater/tower/whatever things that are sprinkled throughout the city to turn them into a sort of "Beacons of Gondor" type thing, as an absolute-last-resort communication method. Mostly jokingly.
No, but if you are providing voice and telephony services that connect to the public switching network, as CenturyLink does, you absolutely have certain reliability requirements for 911 calls being relayed to local answering points.
I await the results of all the inquiries into this to find out what took place here.
This isn't an issue with servicing their direct clients to E911 services, it's cell phone access from various carriers that went out. It's still a bad thing, but not quite the same as what it looks like you're assuming.
No assumptions being made here, just stating matters of fact as to what requirements CTL has for transmitting 911 service, the E911 edit was implied to be supplemental.
It was. In the entire history of electromechanical switching in the Bell System, no central office was down for more than 30 minutes for any reason other than a natural disaster or major fire. There are books about Number Five Crossbar and how it worked, and they're worth reading if you design high-reliability systems.
The only central office downtime from a major fire was in 1975 in New York City, resulting in 23 days of downtime.[1] All the incoming cables and the main distributing frame had to be replaced. The switching equipment, on upper floors, survived and just needed some maintenance and cleaning. That was still a crossbar office; it hadn't been converted to electronic switching yet. Worst disaster in the history of the Bell System. Bell switched to less flammable cable coverings, like plenum cable, after that.
Widespread failure of 911 service suggests an overcentralized architecture. 911 requires a phone number to address lookup, so there's a database involved. Widespread failure indicates this was implemented as a remote query service ("in the cloud") rather than read-only database copies of the directory at each central office.
There's an old AT&T video showing the careful coordination and huge amount of manpower needed to upgrade a single telco switch. They had rows of workers with cable cutters, in order to literally cut the entire service over to the new equipment. Total downtime was just 47 seconds.
(And even during the cutover, they're careful not to interrupt any emergency calls.)
The new switch was connected to each line in parallel with the old switch, but it was able to ignore incoming signals until told to take over. The old switch was hard wired enough that it could not be remotely commanded to have no effect on incoming signals. The line relay attached to each line had to be physically cut out of circuit.
"A History of Engineering & Science in the Bell System: Communications Sciences, 1925-1980".
Here's probably the simplest introduction.[1] When reading this, note phrases like "ORIGINATING REGISTER SEIZES AN IDLE MARKER". Think of that as "originating register asks for an idle marker from the pool of markers". Very little equipment is dedicated to specific lines. Everything is done by requesting a service from one of several identical units. If one of those units fails, the system capacity is reduced, but the switch does not go down as long as at least one of each unit type is still up.
The switch fabric, the actual crossbars, is dumb. It just makes the connections it's told to make.
Shared resources include:
- Originating registers. These provide dial tone and record dialed digits. They parse the incoming number to the limited extent needed to decide when it's finished.
- Markers. The smart part of the system. When an originating register has a full set of digits, it finds an idle marker and sends it the call info. The marker figures out what to do next, in about half a second, and then it's free for another call. Markers tell the switch fabric what connections to make. They're duplicated, and the two halves check each other. If the halves disagree, the marking aborts. If a marker aborts, the originating register tries again with another marker. One retry only. Marker failures also cause data to be sent to a "trouble recorder". As usual, there's more than one of those, and they're "seized" as needed.
- Senders. These send digits from one exchange to the next. They're primitive modems.
- Trunks. Lines between exchanges. Full duplex, four wires.
- Terminating senders. The receive side of senders.
There are also units associated with accounting, coin telephones, routing tables, and other auxiliary functions.
The key takeway here is that there's no single point of failure.
CenturyLink is an incumbent local exchange carrier, as it merged with Qwest, which merged with US West, formed in the 1984 AT&T breakup.
I wouldn't be surprised to find that most emergency call networks rely on the local ILEC, although I would have expected routing to be less centralized.
the decentralized nature of the e911 system is what made it fault tolerant. moving these systems to a more centralized topology is probably going to cost them alot of money in downtime and fines from the FCC in the long run...
Is there any information available about how these systems are typically architected and run as well as what sort of availability they generally achieve?
I can tell you what 911 Call Taking systems look like, they are mostly a call presentation system, with some call handling features. This issue is upstream of any 911 center however.
Interestingly the University of Washington put out an email notice nearly an hour before AlertSeattle sent one. You'd think the city would make it a higher priority to let everyone know that 911 is down. Not sure if they did a text message earlier though, currently overseas w/o service.
It may just be that their email service is not particularly optimized for such an urgent case, and so only got to your email address one hour after the person sent the blast. I've seen CRMs take hours to push out a 40k email blast.
Interestingly the University of Washington put out an email notice nearly an hour before AlertSeattle sent one. You'd think the city would make it a higher priority to let everyone know that 911 is down.
How long do you think it takes to send out a few million alerts versus sending a few thousand?
For an emergency response system, I hope the answer is "about the same time". Or, more precisely, "within the allowed latency window for emergency alerts".
UW's email alert list probably tops 100k. They have like 70k students + staff and there are plenty of former students/staff and associated individuals who are on the lists.
In any case it does seem like the tools that have been built for email marketing are a lot faster and maybe even more reliable than conventional text messaging.
EAS notification would have been virtually instant. But it's supposed to be used only for immediate life-and-death situations (tornadoes, nuclear war, chemical plant explosions).
Amber/cell phone alerts have a lower threshold for pulling the trigger.
I think Seattle eventually activated the local EAS over the radio during the 911 outage, my parents mentioned hearing it, though it was very scratchy and not very intelligible.
> You'd think the city would make it a higher priority to let everyone know that 911 is down.
Should it be? There is an argument that 911 serves as a deterrent and that it would be better if people (in particular, potential criminals, but we can't isolate that portion of the population) didn't find out it was down until it was working again.
It's relatively well known that any situation that produces a predictable breakdown of police response is expired by opportunistic criminals (looting during disasters is perhaps the best known manifestation.)
A publicized 911 outage would be an obvious opportunity.
Most people dont know it. At the least, 911 being down will add delay to the response. Those minutes might be all someone needs for a quick theft, esp cars or burglary focusing ln high-value items.
Could this happen to have been part of the massive CenturyLink/Verizon outage that's been rolling across the midwest the last day or 2? I know it's been a major pain in the butt for me, but I've been super productive at work not having to worry about answering calls.
network engineer for pacific NW ISP here: Yes this is absolutely centurylink related, from what I can observe that it's taken down, and I have my suspicions about Intrado. I think centurylink broke things further trying to fix this morning's DWDM/transport system outage.
My totally subjective opinion: In US/Canadian telecom, any business formed from legacy near-monopoly ILEC (incumebent local exchange carrier) staff, they neither have a clue nor give a fuck. It's a corporate cultural attitude left over from days when ILECs truly were the last-mile monopoly carriers in many cities.
In this case Centurylink is composed of former Embarq, Qwest (USWest) and other ILEC entities.
My opinion also applies to entities like Bell, Telus, Verizon and Frontier.
If you look at the people who run the core routers of their ASes, sure they know what they're doing, but they're so insulated from actually talking to a customer through myriad layers of corporate bureaucracy, that their clue level might as well be irrelevant.
You left off most important part: corruption. Paying politicians to block regulations and lots of competition, esp taxpayer-funded, lets them get away with this stuff. Even more so when one of their own is the regulator.
Was this 10G DWDM gear? Any idea who’s equipment and what root cause was? I’m just curious because I used to work for a vendor that supplied DWDM gear to Centurylink and wonder if this is equipment I designed. 10G DWDM is pretty robust so must have been a pretty massive failure, line amplifier, fiber cut, software upgrade, or something.
"Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership."
This info is related to their much earlier DWDM/transport system outage, which took down a huge number of 10Gbps lit transport circuits starting around 0700-0800 Pacific time on the 27th. The 911 outage for WA began 10+ hours later.
see outages mailing list archives for more details:
Weirdly that almost sounds like some switch getting infected by a worm. eg then attempting to infect any other gear it can (thus the broadcast traffic). :(
Isn't that exactly what cops or others do when they use a Stingray phone tracker and intercept people's mobile phone connections?
I still remember in late 2013/early 2014 when I lived in Kyiv, Ukraine, and my friends at the Maidan protest against the government said everybody within geographic proximity of the protest received SMS messages from an unlisted number telling them their protest wasn't 'legal'. Everybody assumed it was the Ukrainian government security forces using a Stingray or similar device as a MITM between their devices and the network.
If the government is in control of the network or can can order / request the network to do so it would be fairly trivial to send a message to anyone connected to a cell tower.
The thing is if the tower is serving a lot of people you could end up sending the message to people not involved so you could use a IMSI catcher (you can even make them out of a TV Tuner turned SDR receiver[0]) to catch IMSI's in a smaller area or use triangulation to narrow down the pool of devices in an area and then use that list to send SMS from the network itself.
The UK did trails of using location based text messages back in 2013 in order to be used for Public emergency alerts [1]
When you can gain access to the network their is no real need to MITM it, but yeah I believe a sting ray type device could also do the same as they act as a tower and trick your phone into connecting to it.
Cell towers likely don’t run the actual cellular service; they’re just rf points that run the phone applications on top. And if a provider has a fleet of super duper towers that could do it, there’s a certain operations job to be done that needs to be perfectly done. Not trivial. Doable, perhaps, but that would have to be a well rehearsed job for any hope of consistent fleet wide success.
Cell networks (but not towers) effectively do this already - when you dial 911 (or 112), the phone doesn't actually dial the number "911" or "112"; instead it requests a call of type emergency. This is routed by the mobile network itself, according to how it handles emergency calling.
The phone also enters a specific "access class" (access class 10) for emergency calling mode, which lets it join other masts on networks it wouldn't normally operate on. (This is where the message "emergency calls only" originates from)
But if the onward system this "un-numbered" emergency call would be passed onto is unavailable (as it sounds like is the case here) then emergency calls will fail. It might be possible to dynamically reconfigure the mobile network core , but you'd need to configure this on the core, for each and every cell tower or group of cell towers, to route the call to the right alternative lines/number. Whether that system has capacity to handle all these calls is another question.
The incompetence of the Roman government once became so great that they lost the ability to provide basic public utilities like the keeping of time. Is this that for us? How could 911 possibly go down? Crazy.
911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate. There are thousands of (poorly run, uncoordinated) Public Safety Answering Points across the country, and most tend to contract with just the local incumbent telecom, ensuring any time there is an issue with said incumbent, every call to said PSAP fails.
In the context of most of these PSAPs having already moved to VOIP, there isn't a coherent reason why these calls shouldn't go straight from the carrier originating the call to the PSAP, without middlemen like Centurylink standing by to misroute those calls.
I mean if you want to get into the history of any technology you can point out how archaic and horrible it was initially. We're still in that phase with the internet, in 50 years people are going to read about things TCP/IP and Facebook and Comcast and won't be able to stop laughing hysterically at how bad and flimsy it all was.
> 911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate.
I'm always interested in Chicago history, so I went looking, but couldn't find anything at, for example, https://www.nena.org/page/911overviewfacts . Do you know anywhere this history is written up?
And this is why I've made it a habit to save the the local number to the police station where I live, and after moving around a little I seem to have accumulated a few. Always nice to have a backup should you need it.
Not just the police department... Fire, ambulance, poison control, dig safely, local traffic control tower (they're about ~9 miles away from me), pharmacy, and suicide prevention (every fridge needs one of those magnets).
Some airports also have public phone numbers to get weather information, could come in handy to know if you're-for example, living in tornado alley and a system rolls through that knocks out power and cellular service.
If you have a working landline phone, calling a nearby airport could be your only way to know the latest conditions and if it's safe to go outside.
Seems like the government could very cheaply offer an app that is basically just a geolocated lookup for these sorts of things. The app could basically just be a spatialite database generated from a git repo, and it would only have to be updated periodically, and would never go down.
I've always wondered why there isn't a local version of 911. Like 811 connects you to the local police department. Because no one knows the local number. Besides, what if you are traveling? That would also free up the load on 911 operators.
In the UK, ringing 101 gets you through to your local police dept's non-emergency number, and 111 gets you to NHS 24, the national non-emergency medical advice service. Both of those are fairly recent, though - NHS 24 was launched in 2001, and 101 in 2009.
I think the UK definitely benefits from being much much smaller, and the emergency services much more tightly integrated, so rolling out new numbers is not as huge an endeavour as it would be in the States.
311 is increasingly used for local services, but coverage is spotty and usage is inconsistent. In Indianapolis, last I heard, it was only for police use, while in some other cities it’s a general government point of contact.
There needs to be some central or regional oversight body to ensure standards are being met, equitable access is being maintained, etc. A lot of poor areas are undeserved by these patchwork networks and several infrastructure deficiencies can compound. Example: poorly maintained roads leads to potholes, those lead to drivers swerving or getting otherwise surprised leading to injuries, which are insufficiently addressed by local healthcare. Imagine if we had a tyrannical electric utility who cut your power for a bill paid one day late.
At least the systems to notify of the outage sure worked well, at least for me.
Got an EAS alert while watching TV and two emergency alerts on my cell phone (one generic telling me to call local police or fire, and one telling me a specific number to call for my county).
Furthermore, I now think that a call that came in on my landline at 10:42 PM was probably also a notification. The caller ID just said "wireless caller" and I didn't recognize the number so I ignored it, but now Googling that number turns it up as a cell contact number for my county's Department of Emergency Management.
I swear the swiftness with which people swoop into these kinds of threads to suggest crypto/blockchain is bordering on becoming a meme.
If the network is fully down, I don't see how the scenario for this business owner changes with blockchain, they'd have the exact same problem with a register that can't close an ACH: backlogged transactions that need to catch up eventually, meaning some card types are going to decline if the POS can't reach a clearing house utilized by the card vendor.
Can confirm, we've had major issues with calls and call quality across our offices the past couple of days and I have been getting periodic updates from Centurylink about their status in regards to our circuits in my email. Major clients have been effected by this.
Notes from their massive DWDM system outage earlier on the 27th, which preceded the 911 outage:
2018-12-28 13:35:00 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains.
2018-12-28 13:27:30 GMT - The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes. CenturyLink NOC management remains in contact with the equipment vendor to obtain updates as restoration efforts continue.
2018-12-28 11:04:24 GMT - CenturyLink continues to work with the Equipment Vendor to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Client services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes.
2018-12-28 10:05:18 GMT - CenturyLink NOC Management reports steady progression of node recovery and restoral of client services. In addition to the remote node recovery process, Field Operations continue to dispatch and assist the Equipment Vendor with local equipment login.
2018-12-28 08:51:29 GMT - CenturyLink NOC Management has advised that repair efforts are steadily progressing, and services are incrementally restoring. The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels at this time. There have been additional restoration steps identified for certain nodes, which includes either line card resets or Field Operations dispatches for local equipment login, that have impeded the restoration process. Various repair teams are working in tandem on these actions to ensure that services are restored in the most expeditious method available. Restoration efforts are ongoing.
2018-12-28 07:12:32 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. Additional information on repair progress will be available from the Equipment Vendor by 07:30 GMT. Information will be relayed as soon as it is obtained.
2018-12-28 06:00:01 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains.
2018-12-28 04:58:44 GMT - CenturyLink engineers in conjunction with the Equipment Vendor¿s Tier IV Technical Support team have identified the elements causing the impact to customer services. Through the filters being applied and the removal of the secondary communication channels, it is anticipated services will be fully restored within four hours.¿We apologize for any inconvenience this caused our customers. Additional details regarding details of the underlying cause will be relayed as available.
2018-12-28 04:09:31 GMT - The Equipment Vendor¿s Tier IV Technical Support team in conjunction with CenturyLink Tier III Technical Support continues to remotely work to remove the secondary communication channel tunnels across the network until full visibility can be restored, as well as applying the necessary polling filter to each of the reachable nodes.
2018-12-28 02:53:38 GMT - The Transport NOC has confirmed that cooperative efforts remain ongoing to remove the secondary communication channel tunnel across the network until full visibility can be restored, as well as applying the necessary filter to each of the reachable nodes. It has been confirmed that both of these actions are being performed remotely, but an estimated time to complete the activities is not available at this time.
2018-12-28 01:58:56 GMT - Once th...
There was an incident with similar consequences on April 10, 2014. The cause was a programmed threshold being breached and the impact was 6h of downtime.
> Operated by a systems provider named Intrado, the server kept a running counter of how many calls it had routed to 911 dispatchers around the country. Intrado programmers had set a threshold for how high the counter could go. They picked a number in the millions.
I'm really curious if there's some explanation that makes this sound less catastrophically stupid, particularly the part where they picked a threshold less than INT_MAX.
I'm unable to read the article on my phone (ad?) so I apologize if it is answered there, but why is this a long lasting problem? My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order. (I don't want to ignore the pre- and intra- effort by ops, I'm just saying it paid off)
Is this outage just breaking those places that didn't have such things (as a result of insufficient budget or skill)? Is there some now-regrettable contract(s) that prevented that from being in place?
Why is a system that saves lives working without the backup that systems that dont dictate life are? While I understand that govt has both budget and hiring issues, i've worked in state govt and there are definitely some highly competant admins to be found, so I wouldn't assume that is the problem without word from someone with more detailed knowledge into this particular problem.
My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order.
Because your workplaces don't rely on funding from taxpayers.
Since you misunderstood the polite version, here's the reality:
911 failed because there isn't an option to support a competing service, instead people are compelled using violence to pay for a service that doesn't really care about their well being and is usually used to send state thugs to murder people without recourse.
> Because your workplaces don't rely on funding from taxpayers.
Right. Their workplaces probably had tighter and more transparent financial constraints with more oversight, more accountability for failure, and more incentive to do it right.
I have, and been a consumer of them, and have quit them easily.
> I haven't seen any convincing evidence [...]
I guess I could link a page of companies dissolving due to financial mismanagement vs governments or something. The point is not that only one sector suffers mismanagement, it's how that mismanagement is realized/handled/reacted-to, and of course it doesn't apply to every situation. I just get tired of hearing assumptions like a public service is improperly implemented due to funding, especially in this case where I know many of the municipalities are well endowed and should have the redundancy of their systems questioned. Why not ask for convincing evidence for those statements instead?
>Meaning in many cases adverse effects of private sector failure are easier to divorce than public.
Government doesn't tend to provide "easy to divorce services", it's a selection bias problem, not something inherent to public vs privately ran organizations.
Of course it's easier to change office supply vendors than it is to change water companies. Utilities tend to be government operated (or heavily regulated) because their nature makes a competitive market difficult.
>Right, I was, hence the thread context.
Why did you agree with the person you were replying to then? You could have asked for proof in your reply, instead of affirming the comment?
Software is hard, and backbones tend to be mono-cultures. This means that there will be bugs which only become apparent when the full fleet has been deployed.
This is why large phone networks have really long rollout periods - to hopefully catch these things before it affects everything.
This is also why emergency services should not be using cell phones for reliable service during emergencies.
Here's another example from 1990, where the phone network got stuck in a reset loop.
Corporations make similar mistakes with their disaster planning. It doesn't matter if you purchase network connections from two suppliers if they go across the same bridge.
My dad spent his career as a natural gas pipeline technician. For a long time they maintained their own radio towers and VHF or UHF (I forget which) radio network. Each district had their own comms guy who maintained the network, and it was very resilient and reliable, allowing the entire company to be in contact essentially anywhere.
In the early 2000s as a cost cutting measure, the company decided to nix the radios for satellite phones on each truck. Of course this proved to be problematic, as the sat phones often had reception issues. They relied on cell phones as backups, which was also quite foolish as the remote compressor stations had very poor reception. They also had issues where someone would leave a voicemail and they wouldn’t get notified of it for days or weeks due to some issue with AT&T.
After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies and dismantled the infrastructure. My dad retired not long after this, but the “corporate bean counter” trope rang quite true here, and in the long run we were all a little less safe because some executive with no field experience wanted to make a name for themself by saving a little money on something that proved to be mission critical.
yea ... but thats really their purpose. And even in a logical sense the company itself should strive for such measures. Learn where you "can" outsource then do it.
The thorough process and underlying principal isnt bad. Its the execution here that really caused the problem, choosing a provider not as reliable as their own.
Same thing could happen in house. Manager comes in decides to cut cost , sees redundant employees, decides to reduce communications engineers by half. Same problem.
That makes me sad to hear, because not only was this company left with an inferior solution, the man hours and effort that went into the original project (including purchasing part of the spectrum!) was essentially nullified.
It's an alarming trend among new grads as well, who see existing systems as bloated and in need of rewrites. Who would've thought the bloat was kind of important?
But then we wouldn't have single threaded wonders like node.js
It's part of our culture that the young kids dismantle the 'old people stuff' setup their new stuff to only re-invent the wheel again and again, run into the same problems that were solved decades ago to only be repeated again.
Remember how TCL was going to save us all from K&R C? Then it was perl to save us from TCL? Then Ruby? Python? javascript, Go or whatever is the flavour of the month?
I wouldn't argue that bloat is important. I wold argue don't let bean counters make technical decisions.
Unfortunately those bean counters probably got large raises and bonus. Then moved on to their next job proclaiming their victories and got out into higher positions.
BNSF railway still maintains its own point-to-point microwave network and VHF/UHF repeaters on mountaintop tower sites, if you pull the public FCC ULS database data for any western US state you'll see their 6 and 11 GHz band licenses.
And the board let the financial wizard violate the old principle [1] of not taking a fence down until you fully understand the reason why it was put it up in the first place (i.e., don't assume that your predecessors were stupid just because you don't yet understand why they did what they did.)
Really sad results there, throwing away a sound investment & body of know how, for worse results everywhere, and less safety for everyone.
> After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies
This is one of those cases where, if it isn't broke dont fix it was a rule to be followed.
Now its hard for the company to fallback to the old system, since they need to buy another set of frequencies, before they can even plan to rebuild the radio tower network.
Since they had stopped with the good running radio network to cut costs (unwise since it ran fantastic) the chance that they will invest in restarting such a network is not really big, unless they get heavy lossses due to their unreliable current system of modern (but worthless) gadgetry
Great write-up, thanks for the link. TLDR: Race condition when switches reboot during high (read normal) traffic load.
I love the fact the "fix" was to reduce traffic load so that the issue was unlikely to occur and let things go back to normal.
> AT&T solved the problem by reducing the messaging load of the CCS7 network. That allowed the switches to rest themselves and the network to stabilize.
We used a vendor to put up two MPLS drops to every office. different providers, different routes from the building, etc. Turns out, further up the chain, an ISP was buying transit from another ISP, and they all went on the same shared fiber that a nice backhoe operator found for us, about 800 miles away.
In my experience this exists for ISPs, but it's not easily available. I've used ISP brokers in the past and they've had this information in some proprietary systems when I've asked for quotes and proposals.
I worked at two small ISPs, each with a fiber network spanning a few US states. They had their own databases to track this information, but as far as I know there was no central database. One of them used OSPInsight backed by a self-hosted SQL Server database. I don't know what the other place used.
This is very common because of the way transport is built and sold.
Doing real traffic engineering where shared risk link groups do not exist for the active and standby links is a lot of work and, moreover, is hard to keep working as operators change their networks (for maintenance, for other reasons).
On top of the basic difficulty, there are other issues - layering makes this opaque, the service providers have become completely de-skilled, the business model for MPLS is getting killed by Over-The-Top providers who pretend they can give you equivalent service using multiple links and IPSEC tunnels. And so on.
In Australia at least, this is becoming harder as time goes on. Several good alternatives have been purchased by one conglomerate. I know several businesses that did everything right in buying two or even three sets of connectivity from different suppliers, and now what they have are different trading names for the same company providing a service on the same network.
Since the current post adds information, we'll leave it up instead of marking it as a dupe, and have moved most of the non-time-sensitive comments here.
After reading about this earlier today I had the amusing experience of tweeting my local PD asking if "our" 911 was still operational given the situation; apparently it is given their reply tweet.
I do still have the actual "direct" local phone numbers of dispatches for the areas I most frequent.
> Ajit Pai, chairman of the Federal Communications Commission, which regulates and monitors 911 services, said the commission is investigating the outage.
Until they "resolve" the issue without disclosing any details...
Maybe they will open the case for public comments and insights, then “ignore” those comments because of a “DDOS attack”, and then finally not investigate the alleged attack despite earlier on outlining how it was able to cripple public infrastructure.
Could there be a more mismanaged agency than the FCC?
>Could there be a more mismanaged agency than the FCC?
or maybe they're all just as mismangaged, but you noticed them in partiular becaue you are passionate about the FCC's mission?
I'd be willing to bet if the average American interacted with fincen, FDA, IRS etc they'd be shocked at the ineptitude. America has been strangling their enforcement agencies for a long time - one could argue there's a direct path from a poorly funded IRS only taking on "slam dunk" cases and ignoring complex money laundering schemes to Trump's election.
The fact that it takes a special prosecutor (and the subsequent reporting) to dig into these big money laundering schemes only cements your point even more.
If someone neglects to provide their children with food and water, it's rather inappropriate to get mad at the children for their lack of capacity to provide manual labor.
Is this outage for both mobile (cell phones) and landlines?
Does dialing the direct number (XXX-XXX-X911) work?
I've had issues on both when there was no widespread outage to the point I've given up on the utility of 911 and try to put the local police department's number in my phone for emergencies.
Is it me or did TechCrunch recently ruin their mobile experience. I tried to read the article but got so frustrated and distracted by the wonky scrolling I gave up. It’s just awful.
“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” said Pai in a statement. “The CenturyLink service outage is therefore completely unacceptable, and its breadth and duration are particularly troubling.”
In response CenturyLink is now offering a new service, called "911 Fastlane", that guarantees that your 911 calls will get through and will be prioritized ahead of calls from standard CenturyLink accounts. This service is only an additional $10 a month.
Could you help explain why? I really don't see any connection between NN and this story.
I understand that NN is an important issue for the BigCos employing many HN users, but I'm not really convinced we need their talking points to be brought up in every single vaguely network related thread.
The editors of this website remove comments and ban users who put opinions here that go against the mainstream, calling them "incendiary". That way they can ensure this website is basically a monoculture with no "dissidents", the downside is that sometimes low quality comments like that get to the top of the thread.
I don't really care if it's pro or against, both sides are primarily pushed by corporate stooges.
Despite the huge push by ISPs and massive media companies like Google, FB and Amazon, this stuff isn't actually really relevant to normal people.
The conversation surrounding NN has been totally poisoned by corporate interests, even here you constantly see comments suggesting that the internet will be destroyed without NN regulations.
When I read that I always assumed that was a fee that went to the 911 call centers to fund them (paying the people who answered the phones, renting the land they're based out of, etc). Does the entire fee go directly to the phone company?
As the sibling subthread notes, it isn’t just satire (even if the parent intended it as such), and it’s already happening. They already do charge a 911 fee, they just ... don’t use the money to honor the SLA that it promises.
Strange Apple doesn't push their own tech for this. They could include it in Messages and release on Android. For Android there would be a store button where you could see all the new stuff from Apple and buy an iPhone. Would do a service to the 50% of the world's population who's entire community communication network is owned and mined by FB.
227 comments
[ 2.6 ms ] story [ 186 ms ] threadEdit: VOIP indeed. Centurylink outage: https://komonews.com/news/local/centurylink-outage-knocks-ou...
I've consulted with a few rural counties (I don't know a thing about the PNW, is Kitsap a rural or suburban county?) who got their circuits leased to them through a local MSP/IT Shop who had all the best intentions, but overlooked any sort of failover and you end up with an answering center taking calls over a single adtran going direct out to a CLEC.
Just wondering if this is less county refusing to do something, and more like the county implementing services sold to them that probably could have benefited from better planning and integration.
It seems more than likely that Centurylink Sales/Marketing convinced a few counties to use them and then sold itself to other counties as "the" provider for such services.
Not enough, I would wager. It's something I spend a lot of time thinking about, living in one of the largest cities in the country now, with parents who live comfortably in the rural south-but have complained about the internet a couple of times.
To their credit, they'll lose connectivity for hours and not even mind, since mom will just turn on the radio and listen to the local PBS station and tend to her garden while dad goes out to his woodshop.
Others in their community probably aren't as content without connectivity, for such long periods as they are.
There was a always a tension when it comes to outsourcing and considering core competencies in the orgs I saw. Can imagine less appetite for creative solutions when critical things like 911 are involved.
It had more points of failure, instead of one big point of failure. It was also less available nationwide due to the expense.
I await the results of all the inquiries into this to find out what took place here.
Edit: Oh and "But $ISP doesn't offer traditional phone service, they sell VoIP", is as of 2005 no longer a valid excuse for non-compliance: https://www.fcc.gov/document/commission-requires-interconnec...
The only central office downtime from a major fire was in 1975 in New York City, resulting in 23 days of downtime.[1] All the incoming cables and the main distributing frame had to be replaced. The switching equipment, on upper floors, survived and just needed some maintenance and cleaning. That was still a crossbar office; it hadn't been converted to electronic switching yet. Worst disaster in the history of the Bell System. Bell switched to less flammable cable coverings, like plenum cable, after that.
Widespread failure of 911 service suggests an overcentralized architecture. 911 requires a phone number to address lookup, so there's a database involved. Widespread failure indicates this was implemented as a remote query service ("in the cloud") rather than read-only database copies of the directory at each central office.
[1] https://youtu.be/f_AWAmGi-g8?t=110
(And even during the cutover, they're careful not to interrupt any emergency calls.)
https://www.youtube.com/watch?v=saRir95iIWk
Where were the signals going after the cables have been cut?
And then again in 2015.
Also, there was that kid who put together areport of all infra cable runs for a thesis and the FBI confiscated it... https://www.google.com/amp/s/www.nbcbayarea.com/news/local/F...
Here's probably the simplest introduction.[1] When reading this, note phrases like "ORIGINATING REGISTER SEIZES AN IDLE MARKER". Think of that as "originating register asks for an idle marker from the pool of markers". Very little equipment is dedicated to specific lines. Everything is done by requesting a service from one of several identical units. If one of those units fails, the system capacity is reduced, but the switch does not go down as long as at least one of each unit type is still up.
The switch fabric, the actual crossbars, is dumb. It just makes the connections it's told to make.
Shared resources include:
- Originating registers. These provide dial tone and record dialed digits. They parse the incoming number to the limited extent needed to decide when it's finished.
- Markers. The smart part of the system. When an originating register has a full set of digits, it finds an idle marker and sends it the call info. The marker figures out what to do next, in about half a second, and then it's free for another call. Markers tell the switch fabric what connections to make. They're duplicated, and the two halves check each other. If the halves disagree, the marking aborts. If a marker aborts, the originating register tries again with another marker. One retry only. Marker failures also cause data to be sent to a "trouble recorder". As usual, there's more than one of those, and they're "seized" as needed.
- Senders. These send digits from one exchange to the next. They're primitive modems.
- Trunks. Lines between exchanges. Full duplex, four wires.
- Terminating senders. The receive side of senders.
There are also units associated with accounting, coin telephones, routing tables, and other auxiliary functions.
The key takeway here is that there's no single point of failure.
[1] http://wedophones.com/TheBellSystem/pdf/no5crossbar.pdf
Thank you for sharing, that was an enjoyable watch!
Can you recommend such a book?
/edit: i see you answered this for another poster
I wouldn't be surprised to find that most emergency call networks rely on the local ILEC, although I would have expected routing to be less centralized.
https://i.imgur.com/2rlxi7f.png
More info on the outage here: https://www.king5.com/article/news/local/nationwide-centuryl...
How long do you think it takes to send out a few million alerts versus sending a few thousand?
In any case it does seem like the tools that have been built for email marketing are a lot faster and maybe even more reliable than conventional text messaging.
Amber/cell phone alerts have a lower threshold for pulling the trigger.
Should it be? There is an argument that 911 serves as a deterrent and that it would be better if people (in particular, potential criminals, but we can't isolate that portion of the population) didn't find out it was down until it was working again.
A publicized 911 outage would be an obvious opportunity.
Now, if you DDoSed all the local dispatch lines for every police station and sheriff's department in your county, that might be another story...
See the PDF file linked here:
https://www.fcc.gov/document/april-2014-multistate-911-outag...
In this case Centurylink is composed of former Embarq, Qwest (USWest) and other ILEC entities.
My opinion also applies to entities like Bell, Telus, Verizon and Frontier.
If you look at the people who run the core routers of their ASes, sure they know what they're doing, but they're so insulated from actually talking to a customer through myriad layers of corporate bureaucracy, that their clue level might as well be irrelevant.
https://fuckingcenturylink.com/
"Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership."
see outages mailing list archives for more details:
https://puck.nether.net/mailman/listinfo/outages
I still remember in late 2013/early 2014 when I lived in Kyiv, Ukraine, and my friends at the Maidan protest against the government said everybody within geographic proximity of the protest received SMS messages from an unlisted number telling them their protest wasn't 'legal'. Everybody assumed it was the Ukrainian government security forces using a Stingray or similar device as a MITM between their devices and the network.
The thing is if the tower is serving a lot of people you could end up sending the message to people not involved so you could use a IMSI catcher (you can even make them out of a TV Tuner turned SDR receiver[0]) to catch IMSI's in a smaller area or use triangulation to narrow down the pool of devices in an area and then use that list to send SMS from the network itself.
The UK did trails of using location based text messages back in 2013 in order to be used for Public emergency alerts [1]
When you can gain access to the network their is no real need to MITM it, but yeah I believe a sting ray type device could also do the same as they act as a tower and trick your phone into connecting to it.
[0] https://github.com/Oros42/IMSI-catcher
[1] https://www.gov.uk/government/publications/mobile-alerting-t... (link to the info about the trial and the final report)
The phone also enters a specific "access class" (access class 10) for emergency calling mode, which lets it join other masts on networks it wouldn't normally operate on. (This is where the message "emergency calls only" originates from)
But if the onward system this "un-numbered" emergency call would be passed onto is unavailable (as it sounds like is the case here) then emergency calls will fail. It might be possible to dynamically reconfigure the mobile network core , but you'd need to configure this on the core, for each and every cell tower or group of cell towers, to route the call to the right alternative lines/number. Whether that system has capacity to handle all these calls is another question.
Cell towers have virtually nothing to do with routing calls. That’s all centralized in a region with all the towers back hauling.
In the context of most of these PSAPs having already moved to VOIP, there isn't a coherent reason why these calls shouldn't go straight from the carrier originating the call to the PSAP, without middlemen like Centurylink standing by to misroute those calls.
I'm always interested in Chicago history, so I went looking, but couldn't find anything at, for example, https://www.nena.org/page/911overviewfacts . Do you know anywhere this history is written up?
> 1976 Chicago claims to have had "the first enhanced 911 system of any major city" in the United States.
… so I guess in
> > 911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate.
you meant E911 (which, as the linked page points out, is a nebulously defined and variously implemented technology), not necessarily 911?
If you have a working landline phone, calling a nearby airport could be your only way to know the latest conditions and if it's safe to go outside.
This is true of pretty much all types of infrastructure in the USA, from roads to healthcare to electricity.
You mean something like your local, county, or state governments and their agencies?
Got an EAS alert while watching TV and two emergency alerts on my cell phone (one generic telling me to call local police or fire, and one telling me a specific number to call for my county).
Furthermore, I now think that a call that came in on my landline at 10:42 PM was probably also a notification. The caller ID just said "wireless caller" and I didn't recognize the number so I ignored it, but now Googling that number turns it up as a cell contact number for my county's Department of Emergency Management.
Makes you think twice about the cashless utopia the HN crowd imagines.
If the network is fully down, I don't see how the scenario for this business owner changes with blockchain, they'd have the exact same problem with a register that can't close an ACH: backlogged transactions that need to catch up eventually, meaning some card types are going to decline if the POS can't reach a clearing house utilized by the card vendor.
Similarly, for it to make it to the chain only 1 of those 15 confirmations has to connect to any larger network sometime in the future.
https://www.reddit.com/r/networking/comments/a9z6tb/centuryl...
Some areas were fully out. While others (like Denver) saw partial outages/packet-loss as exit nodes depending on the dest were having problems.
Source: "The Coming Software Apocalypse" published by The Atlantic (https://www.theatlantic.com/technology/archive/2017/09/savin...)
walrus01 also linked to https://www.fcc.gov/document/april-2014-multistate-911-outag... in another comment.
I'm really curious if there's some explanation that makes this sound less catastrophically stupid, particularly the part where they picked a threshold less than INT_MAX.
Is this outage just breaking those places that didn't have such things (as a result of insufficient budget or skill)? Is there some now-regrettable contract(s) that prevented that from being in place?
Why is a system that saves lives working without the backup that systems that dont dictate life are? While I understand that govt has both budget and hiring issues, i've worked in state govt and there are definitely some highly competant admins to be found, so I wouldn't assume that is the problem without word from someone with more detailed knowledge into this particular problem.
Because your workplaces don't rely on funding from taxpayers.
911 failed because there isn't an option to support a competing service, instead people are compelled using violence to pay for a service that doesn't really care about their well being and is usually used to send state thugs to murder people without recourse.
HN has so much bias
Wow so even with show dead I can. Barely even read the comments that the HN community has deemed improper.
Awesome sauce!
Right. Their workplaces probably had tighter and more transparent financial constraints with more oversight, more accountability for failure, and more incentive to do it right.
On a more serious note, I haven't seen any convincing evidence that your genetalizations hold true with reapect to public vs private organizations.
I have, and been a consumer of them, and have quit them easily.
> I haven't seen any convincing evidence [...]
I guess I could link a page of companies dissolving due to financial mismanagement vs governments or something. The point is not that only one sector suffers mismanagement, it's how that mismanagement is realized/handled/reacted-to, and of course it doesn't apply to every situation. I just get tired of hearing assumptions like a public service is improperly implemented due to funding, especially in this case where I know many of the municipalities are well endowed and should have the redundancy of their systems questioned. Why not ask for convincing evidence for those statements instead?
???
>Why not ask for convincing evidence for those statements instead
Because I'm not replying to someone making such a statement.
Meaning in many cases adverse effects of private sector failure are easier to divorce than public.
> Because I'm not replying to someone making such a statement.
Right, I was, hence the thread context. The point was wondering why replies aren't there asking, so of course the reply isn't there asking.
Government doesn't tend to provide "easy to divorce services", it's a selection bias problem, not something inherent to public vs privately ran organizations.
Of course it's easier to change office supply vendors than it is to change water companies. Utilities tend to be government operated (or heavily regulated) because their nature makes a competitive market difficult.
>Right, I was, hence the thread context.
Why did you agree with the person you were replying to then? You could have asked for proof in your reply, instead of affirming the comment?
This is why large phone networks have really long rollout periods - to hopefully catch these things before it affects everything.
This is also why emergency services should not be using cell phones for reliable service during emergencies.
Here's another example from 1990, where the phone network got stuck in a reset loop.
https://catless.ncl.ac.uk/Risks/9/62#subj2
Corporations make similar mistakes with their disaster planning. It doesn't matter if you purchase network connections from two suppliers if they go across the same bridge.
In the early 2000s as a cost cutting measure, the company decided to nix the radios for satellite phones on each truck. Of course this proved to be problematic, as the sat phones often had reception issues. They relied on cell phones as backups, which was also quite foolish as the remote compressor stations had very poor reception. They also had issues where someone would leave a voicemail and they wouldn’t get notified of it for days or weeks due to some issue with AT&T.
After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies and dismantled the infrastructure. My dad retired not long after this, but the “corporate bean counter” trope rang quite true here, and in the long run we were all a little less safe because some executive with no field experience wanted to make a name for themself by saving a little money on something that proved to be mission critical.
The thorough process and underlying principal isnt bad. Its the execution here that really caused the problem, choosing a provider not as reliable as their own.
Same thing could happen in house. Manager comes in decides to cut cost , sees redundant employees, decides to reduce communications engineers by half. Same problem.
It's an alarming trend among new grads as well, who see existing systems as bloated and in need of rewrites. Who would've thought the bloat was kind of important?
It's part of our culture that the young kids dismantle the 'old people stuff' setup their new stuff to only re-invent the wheel again and again, run into the same problems that were solved decades ago to only be repeated again.
Remember how TCL was going to save us all from K&R C? Then it was perl to save us from TCL? Then Ruby? Python? javascript, Go or whatever is the flavour of the month?
Unfortunately those bean counters probably got large raises and bonus. Then moved on to their next job proclaiming their victories and got out into higher positions.
Really sad results there, throwing away a sound investment & body of know how, for worse results everywhere, and less safety for everyone.
[1] https://www.chesterton.org/taking-a-fence-down/
This is one of those cases where, if it isn't broke dont fix it was a rule to be followed.
Now its hard for the company to fallback to the old system, since they need to buy another set of frequencies, before they can even plan to rebuild the radio tower network.
Since they had stopped with the good running radio network to cut costs (unwise since it ran fantastic) the chance that they will invest in restarting such a network is not really big, unless they get heavy lossses due to their unreliable current system of modern (but worthless) gadgetry
I love the fact the "fix" was to reduce traffic load so that the issue was unlikely to occur and let things go back to normal.
> AT&T solved the problem by reducing the messaging load of the CCS7 network. That allowed the switches to rest themselves and the network to stabilize.
https://en.wikipedia.org/wiki/Therac-25
All of the unexpected shared infra issues I’ve observed have been down at this lower level that never shows up at the IP layer.
I worked at two small ISPs, each with a fiber network spanning a few US states. They had their own databases to track this information, but as far as I know there was no central database. One of them used OSPInsight backed by a self-hosted SQL Server database. I don't know what the other place used.
Doing real traffic engineering where shared risk link groups do not exist for the active and standby links is a lot of work and, moreover, is hard to keep working as operators change their networks (for maintenance, for other reasons).
On top of the basic difficulty, there are other issues - layering makes this opaque, the service providers have become completely de-skilled, the business model for MPLS is getting killed by Over-The-Top providers who pretend they can give you equivalent service using multiple links and IPSEC tunnels. And so on.
I do still have the actual "direct" local phone numbers of dispatches for the areas I most frequent.
Until they "resolve" the issue without disclosing any details...
Could there be a more mismanaged agency than the FCC?
or maybe they're all just as mismangaged, but you noticed them in partiular becaue you are passionate about the FCC's mission?
I'd be willing to bet if the average American interacted with fincen, FDA, IRS etc they'd be shocked at the ineptitude. America has been strangling their enforcement agencies for a long time - one could argue there's a direct path from a poorly funded IRS only taking on "slam dunk" cases and ignoring complex money laundering schemes to Trump's election.
911 service is VITAL, pay die from heart attacks when ambulances are not sent and people are killed by psychos when cops are not notified.
Make it 100% redundant, all the way. No one single provider should have all of this in one basket.
Does dialing the direct number (XXX-XXX-X911) work?
I've had issues on both when there was no widespread outage to the point I've given up on the utility of 911 and try to put the local police department's number in my phone for emergencies.
In response CenturyLink is now offering a new service, called "911 Fastlane", that guarantees that your 911 calls will get through and will be prioritized ahead of calls from standard CenturyLink accounts. This service is only an additional $10 a month.
Of course, satire.
I understand that NN is an important issue for the BigCos employing many HN users, but I'm not really convinced we need their talking points to be brought up in every single vaguely network related thread.
Don't enjoy capitalism either.
Dont use extremes like I'm using. That isnt allowed.
Wait- 'This is satire'
(FWIW, I don’t think you should be downvoted for objecting to the tangent.)
Despite the huge push by ISPs and massive media companies like Google, FB and Amazon, this stuff isn't actually really relevant to normal people.
The conversation surrounding NN has been totally poisoned by corporate interests, even here you constantly see comments suggesting that the internet will be destroyed without NN regulations.
Someone's gotta pay for those duplicated systems.