29 comments

[ 3.5 ms ] story [ 75.8 ms ] thread
Steeling internet. Neat.
Forging the internet?
(comment deleted)
At least they tried to prepare for this situation. The software they were using was very close to the metal.
Odd how often Verizon's name comes up in these periodic stories about mass mis-routing of internet traffic.
Since they fail to filter these out, perhaps some enterprising person could route the entire internet through someone's closet ISP. Maybe, just maybe, they will then care.
Part of this is Verizon's poor practices, and part of it is Verizon is a major transit provider; if a routing leak affects a large amount of the internet, it's almost certainly because a major transit provider accepted the announcement and propagated it.
Among the many ASes of things they've acquired, Verizon is AS701 / UUNet, which is historically one of the largest and most widespread ASes on the planet. I still think of it as uunet.
I was interested to find out which steel mill, since they are almost non-existent in Pittsburgh these days, but this is about Allegheny Technologies, headquartered downtown, which owns a number of mills and specialty metals businesses scattered around.

There is some steel produced around here still but it's nothing like the old days. The gigantic J&L plant was still in operation when we drove my brother to Carnegie Mellon.

I was also interested to find out which one, given that I'm from Pittsburgh. Turns out that that this is what Allegheny Ludlum is called these days; they pay my grandpa's pension, and several other relatives worked for them for a very long time as well. And today, I work for Cloudflare, though not on any of the stuff that was involved here. It's a small world.
> I was interested to find out which steel mill, since they are almost non-existent in Pittsburgh these days

As evidenced by the smell of sulphur yesterday morning the coke plants down in Clairton are still alive and well!

not the first time and won't be the last time...
It's possible that the Chinese are involved in this with their hacking. The Chinese Communist Party has their spy tentacles in who knows where and I bet the steel mill IT guys haven't figured it out.
If you were the sys admin at that stell mill and you come into the office on that day. What would be different?
You'd essentially be DDoSed. The internet wouldn't work, and your firewall logs would show that you're being flooded with inbound traffic. The firewall would probably be dropping the unsolicited inbound traffic, so internally you shouldn't see any impact.

Unless as part of the misconfiguration that caused this, the firewall also thinks it knows how to route traffic to the affected prefixes, in which case it would be accepting the traffic and routing it, in which case segments of the internal LANs could be flooded too.

More than likely this wouldn't even hit their firewall. BGP misconfigurations generally occur on the router(s), in front of their firewalls, that connect to the upstream providers. Packets come in one connection and go out the other because you're now the shortest path from A to B.
(comment deleted)
Thanks, this is why I’ll never know if I actually fixed the WiFi. My girlfriend says I haven’t.
Tangentially related to Verizon: I lost Verizon cell and cell data service (along with my cable internet through a different provider) this weekend when a construction crew accidentally cut a fiber line.

I had not realized that literally all modern communications were flowing through a single point of failure. It was a surreal experience.

Yeah, redundant lines are expensive.

Even multiple lines to the same DSLAM were considered expensive back in the day... so if you sliced a single line, you're SOL.

BGP filters fix this. Only accept routes from customers for IP space that they control.

I imagine Verizon has some sort of webUI with bad defaults (no filter) so that their helpdesk can setup new customers - Hanlon's razor being what it is.

I remember setting up a T1 in the mid 90's. Our upstream was InternetMCI. There was no Web UI. There was no documentation of any sort. The guy (MCI tech) just had me read him routes over the phone and he announced them. Fun times!
easier explanation is that they were doing troubleshooting and had removed the filter to do that.
In the early days, some upstream ISPs did no filtering. You could announce anything.
so, who's going to get fined?