That may not make any difference, depending on your locality. Unfortunately AT&T infrastructure is still relied on by others such as Sonic, which leases lines from the likes of AT&T fiber, at least in my locality.
Sonic's fiber service uses AT&T fiber in Alameda as well. Just another data point. I have that service. It's been working just fine, though. I haven't noticed any symptoms of flipped bits or anything similar to what others are reporting here.
+1 for Sonic with their own fiber in Berkeley, which was way better than Comcast (previously the only other option) for the same price with no data cap.
It's pretty clear what you're getting, though. If for no other reason than the fact that the resold ATT service costs 2x what "native" Sonic fiber does.
I'm on the preorder list for Sonic, for the past.. almost 2 years now. In the meantime, I'm using resold ATT through Sonic. $10/mo more than ATT directly, but no MITM and no data caps.
Oh man, I thought it was just my crappy old router causing the problems! I've just been too lazy to call them to replace it. I'm going to tweet at them right now.
Edit: Reading the tweet thread, classic interaction: ATT says "please click this to check your connection", guy replies back and says "I think we know more about networks that you do, please get a network admin in here".
What would you know about production networking, Jeremy?
I'm in the same boat. If the dozens of IT pros who are complaining about this can't get AT&T to swap out a single router card, what hope do most folks have?
Fortunately, this malfunction occurred in the SF Bay Area, getting AT&T to fix its network is only a matter of time, sooner or later, the correct person in the Valley would be alerted.
If it happened to the rest of us at somewhere else, we are probably out of luck...
Computer hanging when connecting to random websites some of the time. Apps on your phone just randomly hanging. Streaming services randomly not working.
If you use Twitch you'll randomly have the video break with a connection failure in the middle of stream.
Interesting, I've been having a lot of the same symptoms for months now with my Comcast service on the other end of the country. Comcast refuses to acknowledge there's any problem on their end, but I'm reasonably confident that my home infrastructure is rock solid after double checking every aspect of my network setup.
Default timeout time on firefox is 30 seconds before retrying and a ctrl+shift+r refresh does not refresh the load, so random websites pause while opening and you have no choice but to wait 30 seconds before the page starts loading. (Psst, the default can be changed.)
We are AT&T in SF Bay Area, and our service has been flaking out for weeks. Inability to get DNS, machine claims invalid security certificates, download/upload drop below 1 MbS. All we could get is our modem reset.
I am a new AT&T fiber customer and have noticed that connections to random places seems slow. New connections in general. There seems to be no pattern, even across udp or tcp. I assumed there was just bad connectivity issues. Any more details available on what is happening? Like, what bits are getting flipped?
I might have to go back to the local cable company if this continues.
The ISPs have tried, but they get pushback from local residents whenever they try to install the necessary network boxes at intervals down the street. They had to go out of their way to hide cellular towers as streetlamps to get the 5G rollout to happen; I remember getting two or three notices for this (each from different telcos, so the one street corner has three streetlamps and two traffic lights on one corner).
I suspect if the droughts didn't make people get rid of their green grassy lawns homeowners would be more amenable to seeing green network boxes every few houses. It looks awful in the context of concrete sidewalks, though.
PROTIP: The boxes are an excuse. Building infrastructure would make more people want to move there, and claiming the boxes are ugly is something you can’t disagree with.
The parts of SV that haven't buried their power lines (not too pretty to begin with) have gotten significantly uglier since all of the new cable/fiber/DSL infrastructure has gone in. There are in many cases multiple of these things on loads of poles:
Here in Germany we bury everything. I don't even want to imagine how long it takes to rebuild all of that when some drunkard crashes his car into one of these poles or a lightning strike hits.
I grew up in the Florida Keys where the water table is very close to the surface so it's impossible to bury anything. When I was a young teenager, we would lose electricity all the time from tree branches brushing up against the power lines. They finally solved the problem by putting in power lines on concrete poles that are a few stories up! No more squirrels and tree branches disputing power!
I could be wrong, but I'd worry about earthquakes severing the lines.
This map[0] shows the major faults, but each one of those is a lot of small minor faults that all could snap things and shift. One small fault right by where I grew up is about three blocks long. Last I checked had one earthquake to its name: a magnitude 5.0 aftershock of Loma Prieta
you'd be surprised but it takes quite an impact to disrupt a telephone pole. most crashes leave the pole standing (see examples online), and even when it does topple the wires seem to hold, and the utility company comes in with a fresh pole. the lightning strike factor does suck if you live very very close to the strike and nothing is surge protected. That happened to my house once and fried my xbox and router.
I've seen it happen personally, except it was the support cable to a pole with a transformer. Cable anchor came out of the dirt, pole fell over, sparks everywhere, transformer fluid all over the street, hazmat cleanup crew, big mess.
The parts of SV that have buried their power lines, probably haven't gotten a whole lot of new cable/fiber/DSL than whatever was there when they buried it.
SV is the perfect place to not bury infrastructure. There's no weather to worry about, and mostly people aren't going to shoot down the fiber to try to claim scrap metals.
I live in one of the few cities in the Bay Area where everything is buried, and it's refreshing not to have stuff hanging all over the place and blocking the view. Now, granted, AT&T fiber is slow to come here, but it's hard to know if the fact that the infrastructure is buried is the main reason. They are in some areas of town but not others.
You can bury conduits, by the way, and not cables or fiber directly. This allows you to avoid digging again just to install fiber, for example. There are ways to do it right. And having the infrastructure on poles is not a panacea either. New providers are not necessarily allowed to use the poles. The weather might not be crazy, but the poles are already overloaded and a little windstorm will disrupt electricity, coax cable, or fiber. And of course, those overloaded poles are crazy ugly.
When I used to work in the telecom industry, burying conduit or cable in the ground was anywhere from 3x (bury in some dirt on the side of a county highway) to 20x (directional drilling in a heavily populated city where there's utilities all over the place) the cost per foot compared to hanging it aerially from a labor standpoint.
However, as you correctly point out, there may be restrictions on what you can hang on the poles and where, and oftentimes you'll find poles where it turns out it never should have had the number of attachments it did, but guess who gets to foot a large part of that bill if they want on?
But even then, I've seen absurd lengths gone to in the name of not digging. On Martha's Vineyard, I believe they wound up using a super-special Self-Supporting fiber that could be hung in or near the Power area of a Pole. Yes, that requires a far more trained/well paid worker than normal aerial work. Also, in that region, NESC 250C/D comes into play which makes it even more of a PITA. But it still was far cheaper than putting cable in the ground.
I wonder whether Teraspan or other Vertical Directed Conduit would be a good fit for the bay area (Saw-cut a minimal depth in the street, just lay in a special zip-up conduit for fiber or twisted pair.) If the weather doesn't tend towards large temperature shifts it works well.
Speaking of which, a couple drawbacks worth noting for buried conduit; You have to go out and do your markings, or pay someone to do them for you when a dig request is made, and you have to be ready to handle the repair when someone inevitably forgets to call or the markings are done incorrectly.
Yes, but SV doesn't have alleyways to run the power lines through, so you have all of these ugly wires everywhere and you're not allowed to have big trees in the front yard, since they could interfere with the power lines. It's really ugly and looks like Baghdad after the war.
No, it hasn't made anything uglier - you don't really notice the infrastructure unless you're specifically trying to look for something to get annoyed by.
Maybe you have gotten blind to it? When I first landed in Palo Alto (coming from Europe), I couldn't believe my eyes: 3rd world infrastructure with wires flying everywhere! I most definitely noticed and notice. It's symptomatic of some of the ill that plague Sillicon Valley.
Overhead power / telecom lines (and cable car power lines) have always been a thing in America. Adding fiber and additional infrastructure hasn't really made it uglier. It would be different if we had buried power lines to start with.
Are you thinking about the area just around the SFO landing zone?
I thought that was less of a "won't" and more of a "can't" between the lack of taller building towers and the radio interference zone for the airport means the directional antennas skip a slice there.
This will probably get fixed if 5G becomes a proper thing and there's a lot of micro-cells along the "bay area's biggest parking lot".
NIMBYism at its finest. Cupertino did not allow cell phone towers for a very long time. The only one was an ATT tower on the top of Infinite Loop, right on the Sunnyvale border.
The people who would call their provider about bad cell coverage in their house are the same people that would go to city hall and demand that no cell towers be built in the city.
It’s amusing seeing Cupertino city council transcripts about this because the people show up claiming 5G gives them cancer and the city council desperately tries to get them to use better excuses so they can approve denying it.
Makes a change from the city council members usual practice of denying Vallco permits and claiming Apple employees are hiring prostitutes and molesting high school students. (I did not make this up.)
I know people with Palo Alto Fiber running to their house. They were hosting websites out of their garage years ago (I think archive.org for a while.), but today it's just residential, so it's very much possible.
We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it.
Sonic has their own fiber in some parts of SF/Santa Rosa and you would know if you were on it, all Sonic DSL products are essentially resold AT&T uverse.
In some parts of the East Bay as well, but I can't find exact maps. If they sell you "Fusion IP Broadband", it's rebranded AT&T. If it's "Fusion Fiber", it's Sonic's own.
When AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads.
They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets.
I would probably expect this to be some network card or cable or connector is failing though...
Yes, as of six months ago I had to look up how to opt out of MITMing for a coworker on AT&T Fiber, so that they could use the Internet again when the remote-activated MITM feature inside the modem broke.
The tweet indicates it is at a specific bit position. That isn't symptomatic of a bad cable/sfp/etc. Analog problems like that tend to be more random. Random bit flipping in a fixed bit position is symptomatic of bad ram somewhere, or a router asic gone bad, or various software/config issues.
What I always wondered about this is, unless AT&T has permission from the copyright holder of each and every Web page so modified to distribute the resulting derivative work, how is this practice not criminal copyright infringement?
For the same reason why ISPs are generally immunized from contributory liability that they would otherwise be completely buried in.
Now, if you had written JavaScript to detect and remove these ads, and they went around that, then you might be able to construct a DMCA 1201 claim and sue the ISP for circumventing what is legally considered DRM. Yes, JavaScript can be legally protected DRM. The law doesn't say it has to be good DRM, it just has to have the effect of controlling access to a copyrighted work. And the safe harbors the DMCA provides ISPs wouldn't protect them in this case.
I once had a combination of "dueling" VPN clients on a Windows PC that caused random bit flips. It was awful. SSL/TLS sessions randomly failing. SSH sessions closing randomly. Downloaded files failing integrity checks.
It's an interesting juxtaposition of how tolerant TCP/IP is of random bit flips, yet how intolerant application layer protocols are.
This was used to flip votes in Dominion machines from our greatest president Trump to China Joe, and the stupid Dems forgot to turn it off. Otherwise Trump wins CA easy. Wait till Sidney Powell hears about this.
From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used...
I would use traceroute to find a common bad point for everyone. It is also possible that the networking point where the problem occurs is invisible to traceroute as it could be part of a provider network probably MPLS but at least the common ends of the tunnel would be visible.
The fact that it is a a specific interval indicates a stuck bit in memory.
I think he means if everyone on AT&T experiencing the issue ran trace route to example.com some common hops would emerge, which would be a place to start investigating.
Take a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the problem to find the differences. The finsl set there are good places to start looking. Sweitches and routers along that way could be the cause
It might be better (although harder!) to take the traceroute from example.org, instead of from the clients. Forward and reverse paths often diverge, so it's important to find the path with the error.
If you ping the hops with a large icmp payload, you might be able to observe the flipped bit in the echo reply. That could help isolate which hop it is.
They need to be capturing src/dest IPs as well as ports for AT&T to have any hope of using that data.
Edit to make the comment more useful: If anyone is curious, look up "ECMP hashing." There are probably tons of parallel paths through AT&T's network, and to narrow down to the hardware causing problems, they will need to identify which specific path was chosen. Hardware switches packets out equally viable pathways by hashing some of the attributes of the packet. Hash output % number of pathways selects which pathway at every hop.
Hardware does this because everyone wants all packets involved in the same "flow" (all packets with the same src/dest IP and port and protocol (TCP)) to deterministically go through the same set of pipes to avoid packet re-ordering. If you randomly sprayed packets, the various buffer depths of routers (or even speed of light and slightly different length fibers along the way) could cause packets to swap ordering. While TCP "copes" with reordering, it doesn't like it and and older implementations slowed way down when it happened.
It seems like it, but it's a widespread issue across the SF/Bay Area right now, maybe wider. I've been having it for weeks and exploring it as well. I've even gone as far as ripping the certs off of the router to double check.
You can use ping to more easily hunt these types of issues, for example `ping -A -c 100 -s 1000 -p deadbeef` will show the difference if there is a flipped bit in the payload. You can generate patterns with xxd.
Hardware designers basically started making bad decisions on this issue around the time that VLAN tagging was introduced, as well as harware forwarding of IP packets. When VLAN tags are inserted or removed, the CRC of a packet needs to be adjusted to reflect the inserted, removed and/or modified bytes from the VLAN header. Additionally Both the CRC and IP checksum of a packet needs to be adjusted when TTL is decremented as part of IP routing.
When implementing this functionality, the naive hardware designer will strip the existing CRC from the packet, modify the contents of the packet and then reuse the handy dandy CRC calculation block to place a newly calculated CRC on the packet. Similar choices are made for the adjustment of the IP/TCP/UDP checksums. If any errors are introduced in the contents of the packet by the data path prior to the new CRC is calculated, this results in the CRC being "corrected" to include the erroneous data.
A far more understanding hardware designer will instead calculate how to adjust the CRC by the changes introduced in the packet contents. Sadly, this is far more complicated to get right, and it goes against the drive of hardware designers to reuse blocks of code wherever possible. Every hardware designer working on networking has a block of Verilog or VHDL code to calculate and append a CRC to a packet. Only the most dedicated will attempt to apply only the delta needed to the CRC or checksum.
I'm not a hardware designer but I routinely deal with low level networking shenanigans and I must admit that I never considered that it would be possible to update a CRC without recomputing it fully (unless you were just appending data of course).
This kind of incident happened to me in a system that was supposed to have high availability. We had failovers for hardware, but it seems that a network device that was supposed to have HA (and was set up to pass the functionality to another device in case of failure) did not have ECC memory. One memory bit got stuck at 0 and the event was not detected at network level, as the data was repacked with a "clean" CRC. For some reason the packet headers were not affected by this, maybe because they were kept in a separate memory zone or because of memory alignment. So the device did not report any kind of suspicious activity, no errors in its statistics.
On the application side the effects were quite bad, as the data was mainly XML and, depending on where the bit was flipped, it could impact the data or the XML structure. The data had its own CRC/hash, so the packets were cleanly rejected by the application. Unfortunately the XML library from the message queue engine and the ESB we were using did not like at all when the bit flipping occurred in the XML tags (it seems fuzzing tests were not done at that point) so the message processing got stuck and we kept getting bad messages in the queues. Even worse, the queues could not be cleaned with the normal procedures because the application wanted to first display info about the messages inside - and that failed.
The network debug was non-trivial because of that header consistency - the network devices did not report any kind of packet issues, so we had to sniff the different network segments to identify the culprit. From the application point of view, we had to delete the whole message queue storage to get rid of the bad messages, and let the application handle the rest (luckily it was designed with eventual consistency and self-healing).
Wasted an opportunity to implement code that would detect and handle poison-pill messages. Those will happen in any system where queue is involved and there always needs to be an escape hatch to get rid of them. Deleting the queue is too extreme.
Deleting the queues is an operational decision that I made to be able to put the system back online after the network device was replaced (the important part was the uptime/SLA). From a quick analysis of the logs the percentage of bad messages was ~90% (there was a ~50% chance that the original "touched" bit was 0 so no change was done, but the messages had multiple "touched" bits at fixed intervals).
There was an escape hatch, but the conditions to hit it were a bit complex. Implementing new message filtering of this kind at 2AM while the system was down was not feasible.
I find it amusing that all these multi hundred thousand dollar per year income people working at the forefront of consumer tech don’t have access to a quality symmetric fiber internet connection.
Years ago I had gotten an "ATT fiber coming soon" notice, but still not here. In the end its simply the usual issue with internet access in the US -- ISPs have zero incentive to invest/improve, there's no meaningful competition. With everyone WFH its even worse -- oversubscribed networks are causing trouble all over (esp if you use a cable ISP).
When a city (or someone like Google) brings fiber in a meaningful way, it only then lights a fire under the incumbents' ass. In theory cities with non-buried lines (like Mountain View) may be easier to upgrade; but it still depends on ATT granting pole access (which they promptly slowwalk).
What layer of the stack do we think this is happening at? I seem to recall that GPON has forward error correction, so the same bit flipping in every packet seems unlikely to be caused there. But, I suppose that core routers are complicated enough these days to cause problems like this (custom ASICs doing all the bit moving), and if it affects every customer without common pieces of network equipment in between, it sure sounds like a software problem. More programs, more problems!
It’s certainly a relief to have an apparent explanation for the weirdness I’ve been experiencing! I thought it was my home router or WiFi or something, but then I experienced the same problems at my partner’s house, so I thought maybe it’s just a problem with some websites I frequent or some common internet infrastructure. But we both have AT&T internet, so this must be it!
For me, the problem has mostly manifested as web pages failing to load or appearing to be loading forever. Generally when I refresh the page would load quickly as expected.
I saw the headline and immediately realized I've noticed similar oddness in the Atlanta area recently. Glad to hear I'm not alone. So far I've been blaming it on weirdness from my devices roaming between mesh router nodes, but now I'm going to run the test script overnight and see if anything turns up.
It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS handshake is an error and the connection dies.
Tech support, even when Twitter accounts with a lot of followers message them, is completely useless, they just say that they don't show any outages at your location. They need to be flooded with complaints for someone to look at this, or maybe someone from AT&T is on here that can get it looked at...
I ran the script and it is finding a difference after a while. I have not noticed anything wrong with the network recently, but that of course does not mean there has not been a problem.
Edit: Actually I take that back, I have seen one of the issues where sometimes web sites will not load at random, and a reload fixes the issue.
The files differ after anywhere from 1sec to 5sec for me, and it's always the same character positions, and it always seems to be the same lines, and the same number of lines.
All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses.
The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking.
But if you're one of the 5% you're fucked.
If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.
The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully.
The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensated so that he actually cares about solving the problem) and the charge is refunded only if the problem indeed ends up being on the ISP's side. This should self-regulate as anyone wasting the engineers' time for a simple problem they could resolve themselves would pay for that time.
Who decides if it's the ISP's problem? What if it's "Both"?
About 20 years ago I got escalated to high-tier Comcast support for an issue that turned out to be a little of A, a little of B: Comcast (Might have been @Home, based on the timing) required that your MAC address be whitelisted as part of their onboarding process. Early home routers had a "MAC Address Clone" feature for precisely this reason. At some point, the leased network card got returned to Comcast. Our router continued to work just fine... until about a year later, when the local office put that network card into some other customer's home. We started getting random disconnects in the middle of the day, and it took forever to diagnose, as the other customer was not particularly active with their internet use. Whose fault was it? Ours, for ARP Spoofing? Theirs, for requiring the spoofing?
Wireshark and escalation to a competent tech. I believe they saw weird traffic from their DHCP server, and we were able to attach an ethernet hub (Not switch, a 10-Base-T Hub that repeated the signal on each port) along with a laptop that was running Ethereal (Before the name changed! How long ago that was now) and see the arp packets fighting.
> I believe they saw weird traffic from their DHCP server, ...
That makes sense.
When the cable modem issued a DHCP request, the CMTS would have been configured to insert some additional information (a "circuit-id") into the DHCP request as it relayed it to the DHCP server.
The short version is that the "competent tech" looked at the logs from the DHCP server, which would have showed that the "same" cable modem (i.e., MAC address) was physically connected to either 1) two different CMTS boxes or 2) two different interfaces of the same CMTS.
How would one cable modem be physically present in two different locations at the same time? Obviously, it wouldn't.
At that point, either 1) there are two cable modems with the same burned in address or 2) one of the two cable modems is cloning/spoofing its MAC address. Which one of those is more likely?
(If you're interested in the details, try "DHCP Option 82" as your search term.)
I realize this was written from a position of frustration (which I share) at getting run around by customer support, but I'd reconsider the blanket characterization of tech support staff as "idiots": they're doing a high-throughput job following a playbook they're given with, as you identify, no incentive---it's probably less about personal motivation than the expectations that are set for how they perform their job---to break rules to provide better customer service to people with 5% problems.
Good point. The managers would probably give negative feedback to people that took more time on their calls in order to try to help the customer better.
+1 the real idiots here are att mid-upper management that setup this process and also have zero monitoring for packet loss/bit flips apparently so that they have an outage for weeks now. Support techs have no training nor tooling to debug this issue.
I cope with level 1 support by remembering we have a common goal: to stop talking to each other as quickly as possible. The tech just wants to close the case and I want to talk to someone else who can actually help me.
To be fair, the vast majority of technical support calls likely require only customer service rather than technical skills. Anyone with technical chops probably wouldn't last long in that environment.
A friend of mine used to work tech support, and said that from her perspective, a lot of the effort was trying to counter-outsmart customers who thought they were too smart.
For instance: "This might just be a loose cable, can you unplug each end of it, and plug it back in?" invariably elicited an "I've already done that", or a brief pause followed by "okay, there, I did it, do you see it?". Lies, often.
But: "Alright, I want you to try turning the cable around for me, yeah swap it end for end. Sometimes you get a connector that just doesn't fit quite right, but it works the other way around and it's faster than sending a tech to replace it", would often get a startled "Oh! I hadn't thought of that, one moment..." and then the customer actually DOES unplug the thing, and what do you know, they click it in properly this time.
I worked for a call center and there was a girl there I didn't think was good at her job.
Then one day, she tells me that she tells people to unplug the power cord, cup the end in their hand, and then plug it back in.
Suddenly, I really liked her. It's a genius move that makes them think they've done something obscure, but she really just wanted them to actually check the cable.
A huge percentage of companies have their help desks and customer support outsourced to TCS or Cognizant or Accenture, etc.
Everything they do is driven by metrics, and their contracts are written around KPIs like maintaining a ludicrously short average time to answer, short handle times, and open/closed ticket ratios. If they do not hit these metrics, then the outsourcing company owes service credits. The incentives do not align with making customers happy and solving their problems. Everything is geared towards deflecting users with self-help options, simple scripts for the agents, and walking the line of hitting those metrics with the fewest number of warm bodies.
This blows my mind, because why would a customer come back when they're treated like that? Oh right, because they don't have anywhere else to go.
For the last several years, I've gotten my mobile service through an MVNO named Ting. On the rare occasion that I've needed to call support, there's no IVR, just a human who typically answers on the second ring. They speak native English, and have never failed to solve my problem either immediately or with one prompt escalation.
They're so jarringly competent I wonder how they still exist, if being obnoxiously incompetent is apparently a business requirement.
I worked for a computer support once, and just as I was hired, they went from (IIRC) 6 weeks of training to only 2 weeks. Nobody came through that knowing any more than they started with in regards to fixing computers.
Luckily, I already knew how to fix them. I found the job to be a cake walk and quite liked helping people. But I had to listen to people around me fumble through it.
It was frustrating at the time, but my favorite thing that happened was that I was admonished twice for having average call times that were too low. To them, that's a warning that someone is just getting people off the line without fixing their problems.
They monitored calls and they said I'd never received a complaint, but the system would keep flagging me for low call times so I had to artificially raise them. They suggested that I have a conversation with the clients.
I didn't, and I didn't stay there much longer, but it was quite a crazy situation. But I also felt much less pressure to handle calls quickly after that, too, which was nice.
I used to play counter strike/Starcraft in my middle school years. I pretty much figured out I had consistent packet loss with a simple ping test. I was on the phone with Time Warner every other day for months. They kept sending the regular technicians, at one point ripping out all of my cable wires in the house to see if it fixed the problem. Nothing worked, I kept calling, at this point I had the direct number to level 3 support. They saw the packet loss too. Finally, after two or three months they send out a Chief Engineer. Guy says I’ll look at the ‘box’ on one of the cable poles down the block. He confirmed something was wrong at that source for the whole area. Then it finally got fixed.
Took forever dealing with level 1 support, and lots of karenesque ‘can I talk to your supervisor please’, but that’s literally what it took.
So yeah, if you want stuff like this fixed, stay professional, never ever curse, consistently ask to speak to the supervisor, keep records, and keep calling.
Small shout out to the old http://www.dslreports.com/ for being a great support community during the early days of broadband for consumer activism in terms of making sure you got legit good broadband.
I had similar issues for a long time. Dealing with my ISP’s support was really frustrating. Not once did a 2nd or 3rd line technician get in touch with me to acknowledge that they had done any kind of investigation and analysis of the intermittent issues that I kept experiencing. The ISP did send out a guy that replaced the optical transceiver in my end, but to me it just felt like a wild guess and not really something that they did because they had any specific reason to believe that the transceiver was actually faulty. It didn’t help.
I ended up just cancelling the service and signing up for one of their competitors instead.
And conversely its always surprising and disarming when you call a company and actually get through to a knowledgable employee. I was so surprised to hear “thats a firmware bug we know about and there is no update yet” about my router issue that I forgot to be mad at the company for not caring my router is broken.
I had a problem with a PowerMac G3 back in the day and I somehow managed to escalate up to tier 3 which is to say an Apple HW engineer. He was brusque bordering on rude, but he immediately recognized it was a problem with an undocumented jumper setting on the motherboard and solved my problem inside of two minutes. It definitely increased my customer satisfaction.
A few years back, I called up our local newspaper to start a subscription. Called the number on the website, and a real human person answered the phone. I was so surprised that it wasn't at least an initial phone tree that I actually stumbled and had to apologize and explain myself.
Way back in the day I worked on equipment that straddled telco circuits. T1s, E1s, DS3s, OC whatever. Companies paying big money for those circuits.
Anyway I was told on more than one occasion by different telcos that the standard operating procedure for many techs was to take the call, do nothing, and call back 20 minutes later and ask if it looked better because ... often enough it did.
When I started my job as an IT director 10 years ago I was in the customer support room and there was one guy who was known for solving all the hard problems. I was standing behind him when he was talking a call. He patiently listened to the client, then loudly typed random stuff on his keyboard for twenty seconds or so, making sure the client could hear the frantic typing, sighed, and then asked “Is it better now?”
> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.
When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path worked.
Someone who has taken the time to report an issue is probably one of hundreds or thousands who had an issue and didn't think it could be fixed and shrugged it off. We certainly can't fix everything, but weird network shit like this can be fixed, and it's worth escalating, because when you get it fixed, you can also figure out (hopefully) how to monitor for it, so it doesn't happen again.
OTOH, I didn't work for the phone company. We don't care, we don't have to, we're the phone company. https://vimeo.com/355556831 (sorry about the quality, I guess internet video was pretty lowdef in the 70s :P)
Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process.
The most memorable one:
Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they answered, the party on the other end didn't seem to hear them. They seemed to be talking to _someone_, but not the party they were connected to. Eventually they hung up. Sometimes, a customer would place a call, and be on the other end of the same situation -- whoever answered would say hello, but the two parties didn't seem to be talking to each other. Off into the void. They'd try again, and it would work, usually, but repeats weren't uncommon.
So everyone's looking at system logs and status alarms and stuff, and what else changed? There were two new racks of echo-cancellers placed in service last night, could that cause this? Not by any obvious means, I mean e-cans are symmetrical and they were all tested ahead of time. There was a fiber cut out by the railroad but everything switched over to the protect side of the ring OK, didn't it? Let's check on that. Everyone's checking into whatever hunch they can synthesize, and turning up bupkus.
Finally around lunchtime, one of the techs bursts into the ops center, going "TIM! I GOT ONE I GOT ONE IT'S HAPPENING TO ME, PATH ME! okay look I don't know if you can hear me, but please don't hang up, I work for the phone company and we've got a problem with the network and I need you to stick on the line for a few minutes while we diagnose this. I know I'm not who you expected to be talking to, and if you're saying anything right now, someone else might be hearing it, but that's why this is so weird and why it's so important YEAH IT CAME INTO MY PERSONAL LINE and that's why it's so important that you don't hang up okay? I really appreciate it, just hang out for a few, we'll get this figured out..."
Office chairs whiz up to terminals and in moments, they've looked up his DN and resolved it to a call path display, including all the ephemera that would be forgotten when the call disconnects. Sure enough, it's going over one of the new e-cans. Okay, that's a smoking gun!
So they place the whole set of new equipment, two whole racks of 672 channels each, out-of-service. What happens when you do that is the calls-in-process remain up, but new calls aren't established across the OOS element. Then you watch as those standing calls run their course and disconnect, and finally when the count is zero, you can work on it. (If you're doing work during the overnight maintenance window, you're allowed to forcibly terminate calls that don't wrap up after a few minutes, but that's verboten for daytime work. A single long ragchew is the bane of many a network tech!) The second rack was empty of calls in _seconds_, and everyone quickly pieced together what that implied -- every single call that had been thus routed was one of these problem calls where people hang up very quickly. This thing had been frustrating hundreds of callers a minute, all morning.
With the focus thus narrowed, the investigation proceeded furiously. Finally someone pulls up the individual crossconnects in the DACS (a sort of automated patch panel, not entirely unlike VLANs) where the switch itself is connected to the echo-cancellation equipment. And there it is. (It's been too long since I spoke TL1 so I won't attempt to fake a message here, but it goes something like this:) Circuit 1-1 transmit is connected to circuit 29-1 receive, 29-1 transmit isn't connected to anything at all. 1-2 transmit to 29-2 receive, 29-2 transmit to 1-1 receive. Alright, we've got our lopsided connection, and we can fix it, but how did it happen in the...
Cool story, thanks! Perhaps you can solve a mystery phone hiccup that happened to me a few years ago? I called a friend (mobile to mobile if it matters) and, from memory, about 20 minutes into this call I get disconnected, _but_ I instantly end up on a call with an elderly stranger instead, who seemed pretty irritated she was now on the phone with me. I was surprised enough that she hung up before I could form a coherent sentence to explain what had happened so I've no idea if she was trying to ring someone or if the same thing happened to her or if she'd dialed my number by accident. From what I remember it seemed like she was also already mid-conversation as well though.
Not OP but it sounds like the echo cancellers were fine, the interconnect to the switch was misconfigured. Rather than sending both channels of audio to opposite ends of the same call, one channel got directed to the next call.
The funny thing is that if everyone played along they could have had a mean game of telephone going.
They passed testing because they had each been individually crossconnected to a test trunk, and test calls force-routed over that trunk. Then to place them in service, the crossconnects were reconfigured to place them at their normal location in the system. The testing was to prove the voice path of each DSP card, and that those cards were wired into the crossconnect properly.
All that was true, the failure happened when it was being taken out of testing config and into operational config. Either nobody considered that that portion could fail, or the urgency to add capacity to a suddenly-overloaded network meant that some corners were cut. (Marketing moves faster than purchasing-engineering-installation...)
Oh, and as to the point about keeping the call path ephemera. Yeah probably, but in a server context, that'd be akin to logging the loader and MMU details of how every process executable is loaded and mapped. Sure, it might help you narrow down a failing SDRAM chip, but the other 99.99999% of the time when that's not the problem, it's just an extra deluge of data.
As I recall, the cross-connects were done at the DS1 level, and an individual card handled 24 calls. These are hazy, hazy memories now; this took place in 2004-ish.
Amazing story! My step dad worked night shift at AT&T back in the 80’s and ran the 5ESS. He took my brother and I in for a tour one night. Thinking back on it now it was a lean crew for the equipment they were running. Rows and rows and rows of equipment. I don’t remember closed cabinets, mostly open frames moderately populated. I’ll never forget he showed us some magnetic core memory that was still mounted up on a frame in the switch room. Huuuge battery backup floor as well.
He loved all of that stuff, absolutely hated when everything went to computers. Quit and became a maintenance man at a nursing home, commercial laundry repair guy then finally retired this year in his late 70’s (due to Covid) after working maintenance at a local jail.
I believe the #5 ESS machine itself is always in closed cabinets, so it's likely that what you're remembering was the toll/transport equipment, or ancillary frames. Gray 23-inch racks as far as the eye can see!
Depending on how old that part of the office was, they were likely either 14' or 11'6" tall with rolling ladders in the aisles, or 7' tall and the only place they'd have laddertrack was in front of the main distributing frame.
As for magnetic core, if you could see it mounted in a frame, what you probably saw was a remreed switching grid, which is a sort of magnetic core with reed-relay contacts at each bit, so writing a bit pattern into it establishes a connection path through a crosspoint matrix. It's not used as storage but as a switching peripheral that requires no power to hold up its connections. (Contrast with crossbar, which relaxes as soon as the solenoids de-energize.)
You’re definitely on to something. This image from Wikipedia for the #1 ESS fits very well into my fuzzy memory, especially those protruding card chassis:
I just remember thinking it looked awkward getting to the equipment under them.
I don’t know if the ‘5E’ as he called it was actually in operation yet, he ended up moving us all out of state to take a job developing and delivering training material for it...I think that’s what finally broke him lol. Hands on kinda dude.
I’ll have to hit him up later today to see if he remembers ‘remreed’ (he will). Thanks for the info!
Yup, the #1 used computerized control, but all the switching was still electromechanical, so it sounded like a typewriter factory, especially during busy-hour.
At night, traffic was often low enough that you could hear individual call setups and teardowns, each a cascade of relay actuations rippling from one part of the floor to another. The junctor relays in particular were oddly hefty and made a solid clack, twice per call setup if I recall correctly, once to prove the path by swinging it over it to a test point of some sort, and then again to actually connect it through. On rare occasion, you'd hear a triple-clack as the first path tested bad, an alternate was set up and tested good, and then connected through.
Moments after such a triple-clack, one of the teleprinters would spring to life, spitting out a trouble ticket indicating the failed circuit.
The #5, on the other hand, was completely electronic, time-division switching in the core. The only clicks were the individual line relays responsible for ringing and talk battery, and these were almost silent in comparison. You couldn't learn anything about the health of the machine by just standing in the middle of it and listening, and anyone in possession of a relay contact burnishing tool will tell you in no uncertain terms, that the #5 has no soul.
Theres a telco museum in Seattle called the Connections Museum, it has working panel, #1 crossbar and #5 crossbar switches and a #3ESS they are working on getting running again.
This feels like the kind of anecdote I'd overhear my paint-covered neighbor Tom telling my dad when I was 10, and my dad would be making a racket over it, really bent over twice. I'd always be like, "what's so funny about that?" But you get older and you realize not many people tell _actually_ interesting stories, so I guess you do what you can to make them want to come around and tell more.
Thank you for sharing. And for helping the phones just work, so we can complain so much when they don't :)
Have you heard any of Evan Doorbell's telephone tapes[1]? It's a series of recordings mostly from the 1970s, but with much more recent narration, exploring and sort-of documenting the various phone systems from the outside in. Might be interesting to see what they figured out, and what they didn't :)
I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it.
It had happened before and then magically fixed itself a few days later.
One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basically cabling guys, better for finding a bad cable than debugging a packet loss. The problem for me was that my ipv4 static ip addresses would not receive traffic.
I was at wit's end after a week and I debugged the thing myself. By looking at EVERY bit of data on the router, I found mention of the blocked packets in the firewall log. I would clear all the logs, and found even with the firewall DISABLED, the firewall log would see and block incoming packets I was sending using my neighbor's comcast connection.
I called AT&T, but this time mentioning "firewall is completely off, but packets are blocked by the router and showing up in the log" was concrete enough for them to look up a (known) solution.
The fix was to disable the firewall, but to enable stealth mode. wtf?
To be clear, this was a firmware bug, and caused dozens of calls to AT&T, lots of heartache and finger pointing always in my direction.
I should also mention at the start of this fiasco, I checked the system log and noticed they pushed a firmware update to the modem at the time the problem started. Strangely after one call to the agent, that specific line disappeared out of the log file, but other log entries remained. hmmm.
since then, they basically screw up my modem every month or two - they push new firmware and new "features" appear (like the one that sniffs and categorizes application traffic like "youtube" and "github"). It also helpfully turns wifi BACK ON when I had disabled it. I immediately go turn if back off and then they immediately send me a big warning email that my DSL settings have been changed.
The 5268ac pace router is the worst ISP provided router I've ever had, and I've been an Xfinity/Comcast customer, and I've even had a connection in Wyoming. I detailed my experience with it in a review of a third-party router, and found numerous issues along the way [0]. My favorite is that DMZ+ mode, which is what they offer instead of a traditional DMZ mode, just has some weird MTU issue that leads git and other services to break horribly when running behind a third party router. The solution? Don't use DMZ+ mode. Instead, put the router into NAT mode, and then port forward all of the ports to one private IP address. Bonkers. This is sold as an official-looking solution on the AT&T website for a "speed issue." [1]
This is all because AT&T believes that the edge of their network is not the PON ONT/OLT, but rather, the router they issue you. If you want to be on their network, you have to use their router as some part of the chain.
My latest discovery is that in doing this, the router can actually get super hot operating at gigabit speeds for extended periods of time. When this happens, it magically starts dropping packets. Solution? Aim a fan at the router so it has "thermal management."
Total. Garbage. I'd switch to Spectrum if they had decent upload speed, but alas, they don't in my building.
Apparently you can extract the 802.1x key from the router and then use your own router, and someone even has a script to MITM the connection between the router and ONT.
If you have some time, you can MITM the 802.1x auth packets [1] and use a less crappy router. I run this with a VyOS router and the same 5268ac that you have, but it works with things like Ubiquiti routers too. The only catch is you need three NICs on your router, but a cheap USB 10/100 one will do for the port that connects to the 5268ac.
Another option is getting the 802.1x certificate out of a hacked router, but it's not possible as far as I know on the 5268ac. You could buy a hackable ATT router but they're not cheap. Some sellers even sell the key by itself.
Mysteriously, doing this fixed an issue I previously had where SSHing into AWS would fail.
BGW320 is the new model, which I had installed about a month ago. It isn't a simple swap, as it uses a SFP module combined with the modem's internal ONT instead of a separate ONT, so I've heard it's only used in new installations. More about it: https://www.dslreports.com/forum/r32605799-BGW320-505-new-ga... (although theirs says 1550nm while mine says 1310nm)
However, it has 5Gbit Ethernet, hasn't re-enabled WiFi on automatic firmware updates, and has only screwed with my IP Passthrough configs once which was resolved with a router reboot. (that was possibly my router's fault, it seemed like it was unable to fetch a new DHCP lease)
That is absurd that AT&T requires the use of a rented gateway for U-verse. I've never had an issue with a another provider refusing to support off the self hardware before, including with a DSL provider, multiple cable companies, and FiOS (Ethernet on ONT).
> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.
... only if you want people to actually be able to escalate. Suspect AT&T are going to lose a lot less money over this issue than hiring an extra high quality support person would cost
That’s why you need to sign up for CSA Pre™. Get preauthorized for instant escalation on customer support calls.
All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc.
CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.
I've had this exact experience, except that it wasn't a dream.
Back in 2010 I had a weird issue where my cable connection would sometimes completely block the connectoin right after the DHCP response (we had dynamic IPs back then). This would go on for a couple of hours, until the IP lease expired, then my connection would come back. Luckily, I was running an OpenBSD box as my router which allowed me to diagnose the problem. But it was also impossible to explain to the servicedesk employees.
One evening it happened again, and I called the servicedesk, totally prepared to do the 'yes I have turned it off and on again' dance. But to my surprise the employee that I got on the phone was very knowledgable and even said that it was very cool that I had an OpenBSD box as a router. He very quickly diagnosed that someone in my neighbourhood was 'hammering' the DHCP service by not releasing his lease (a common trick to keep your IP address somewhat static). This caused a double IP on the subnet, and the L2 switch to block traffic to my port.
He asked me "do you know how to spoof your MAC with an OpenBSD box?". Then I knew this guy was legit. He instructed me to replace the last 2 bytes of the MAC with AB:BA (named after the music group). They had a separate DHCP pool for MAC addresses in that range. If they ever saw an ABBA mac address on their network, they knew it was someone who had connectivity issues before.
The problem was immediately solved, and I had a rock-solid internet connection for years, with a static IP!
I ended up chatting about networking and OpenBSD a bit, before I (as humble as I could) told the guy I was a bit flabbergasted that someone as knowledgable as him was working on the servicedesk.
It turned out, he was the chief of network operations at the ISP (the biggest ISP in my country). He was just manning the phone while some of his colleagues from the servicedesk were having diner.
100% you’re on point here, but there’s one problem that happens when they do add “is this a 5% problem”: eventually (and I’d say pretty quickly), the public gets wind of “if I say the right things to get marked as a 5% problem, I get an automatic escalation to someone who knows more.”, and suddenly you get a big chunk of level 1 calls in to the upper tiers.
For evidence, see: “Retention departments give you the best rates if you threaten to cancel” (which then caused companies to have to rename the retention departments and change the policies.
My AT&T bay area fiber DNS has been garbage since the day we got it. I've had to forcibly update every device we own to Google DNS as well as put another router in between our devices and the AT&T one since AT&T doesn't let you actually change those settings on your device.
Why wouldn't the first conclusion be that the example.com service was flipping bits? Load balancing TLS requests to a different front-end pool is very common.
Has anyone claimed to see these flipped bits on a domain other than example.com?
This is a pretty useful piece of debugging here. It might be worthwhile to try to get EdgeCast involved, as it could be a broken thing between AT&T and EdgeCast.
Given that (it seems) only AT&T customers are complaining, and (it seems) it only affects servers on EdgeCast.
Hardware that forwards packets usually forwards IP packets, it doesn't care about TCP checksums and doesn't recalculate them. TCP stack in your OS is the one that does that.
It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packets.
Switches (L2 devices) recalculate Ethernet CRCs (L2 error detection codes), and routers tend to calculate TCP/UDP checksums (L3 error detection codes) and everything below. I've seen exactly this issue with switches that have bad RAM before, and I assume that they just have a router with bad RAM (and without ECC RAM, which looks embarrassing).
Checksums are often calculated in hardware on the NIC. I have personally seen a network card send packets with corrupted data (corrupted by the network card itself) and valid TCP checksums, computed on the corrupted data.
Unfortunately TCP checksums are hot garbage given switch ASIC design. They are a 16 bit one's complement sum over a packet. If you get two bit flips in the same offset % 16, you can pass a checksum.
The problem is routers slow down the high speed serial signals from fiber to by splitting the bits over a large number of slower speed signals internally. Often those wider busses are a multiple of 16 bits. For example, one ASIC I know of moves things around in 204 byte chunks. (Might have been 208, been a while.) Anyway, the problem is that if there is a defect in one of those parallel elements it will always flip bits in the same offset position mod 204 bytes, which is the same position mod 16 bits. If the hardware is degraded enough, it can end up flipping two bits in the same position, and that has a fairly good chance of passing the checksum.
Ethernet has proper CRCs on packets, which is a lot less vulnerable to shenanigans like this, but unfortunately those can end up being checked on the way in, discarded, and then re-generated on the way out of a router. If anything is corrupted in the middle of the switch ASIC, nothing notices and it passes along. I once helped troubleshoot an issue in our network where a BGP packet was corrupted in this way. The flipped bits ended up causing a more specific route to be generated, and we had the world weirdest BGP route hijack within the bounds of our own data center.
Could be crappy/buggy middle boxes. Especially if they’re inspecting packets or messing with SSL.
My ATT modem (Arris bgw-210-700) may have gotten a firmware upgrade recently as I found some settings I shut off got re-enabled. I use it for VDSL but the same model family is used for ATT fiber.
TCP checksum is simples one's complement checksum. So if two bits are flipped in the same position in two different 16 bit word, they cancel each other out. If you look at all the diffs posted, they differ in even number of lines.
In the above linked gist, how we we know it's a low level router bit flip instead of some code / programming error in their MITM / dns / JavaScript injection tomfoolery?
Much scarier when you actually see it with your own eyes:
$ diff example-*
4c4
< <titde>Example Domain</title>
---
> <title>Example Domain</title>
11c11
< backoround-color: #f0f0f2;
---
> background-color: #f0f0f2;
23c23
< box-shadow: 2pp 3px 7px 2px rgba(0,0,0,0.02);
---
> box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
41c41
< <p>This domain is for use in illustrative examples in documents. You may usm this
---
> <p>This domain is for use in illustrative examples in documents. You may use this
I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?
I don’t have numbers off hand, but from a feeling I have from memory, I would think it is extremely unlikely that TCP checksums are consistently failing to trigger retransmission. Someone must be altering packets along the way.
Checksums are also often recomputed on transit if the packet is intercepted, e.g. to limit TTL, unflag odd/unused TCP features, that kind of ISP-ish preening. So if it was a software error or even a hardware problem in the right (wrong) spot it’s possible to get this kind of corruption without retransmits.
In that case, won't there be significant packet loss causing throughput to be very slow? I don't know if this is possible without something messing with TCP headers.
The checksum should be checked by your computer. So somehow the packet is being repackaged with the correct checksum, but for the wrong data. In other words, when your computer checks the checksum, it matches. Another possibility is that somehow only errors that result in the same checksum are being generated.
Except it was trivial to reproduce with the script on non-Apple devices, and people in one of the many Twitter threads surrounding this showed that on their Mac there was MANY tcp retransmits due to invalid checksums, and the bit-flipped packet did have the correct checksum.
OK, that tweet does show the checksum is OK. I didn't see a whole lot of tcpdumps, so had to go with what was reported in the thread (I tried to reproduce with a few people, but my server wasn't in the broken path, so I couldn't get a lot of real data).
That tweet in particular doesn't show any retransmits.
tcpdump/wireshark gets a little hard to read at times; especially when the packet dump is a lie: all those packets marked red for bad checksums are from the dumping machine, and the checksums are wrong because the NIC is filling them in, and the capture interface doesn't get to see what they are). Perhaps the other people in the thread who said mac os was ignoring bad checksums were also confused; or perhaps it does ignore bad checksums, it's pretty bad at networking (it can't handle a synflood in 2020 because it's got synhandling code from 2000)
There’s no proof of this. And what possible reason would macOS have for not checking the checksum? Although the checksum is weak it presumably catches at least some corrupt traffic. Do you really think Apple would just skip the TCP checksum and make its network performance less reliable when they have already implemented (or maintained if it came from BSD) the rest of a TCP/IP stack, which is vastly more complex, just because its developers are lazy?
There were reports in the twitter threads that macs were ignoring the bad checksum (and I thought I saw confirmations in this thread, blaming a driver). Since I don't have a mac anymore, I couldn't confirm or deny; and since I don't have any equipment with the bad equipment in the path, I couldn't get a tcpdump, and I hadn't seen any on twitter.
I wouldn't have expected Apple to purposefully break the checksum, just like they I don't think they purposefully have no synflood protection because they pulled the TCP/IP stack at the turn of the century and never pulled it to get the many many many upgrades from upstream (although, they did add on MP-TCP, so there's that). I wouldn't be surprised if tcp checksums had stopped working ages ago, possibly because of a aggressive driver, and nobody noticed. Kind of like how if you spawn a few thousand threads that just sit around sleeping, it will delay watchdog kicks and the kernel will panic. (also, from reports on here, not personal experience)
It is extremely common for hardware to be configured to ignore checksums. ("A packet with a bad checksum would have been dropped before it got here. Our cabling is too short to drop bits.")
> presumably because a corrupted packet in a TLS handshake is an error and the connection dies
Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and your crappy Internet mangling the data in transit, in either case the TLS protocol design says to "alert" bad_record_mac which your browser or similar software will probably treat as a failed connection, even if it happens mid-way through an HTTP transaction.
Because TLS guarantees integrity even if your fiber is a complete shit show, any TLSPlaintext records which do get from one end to the other are guaranteed to be as intended.
Help me understand, are you saying only HTTP bits are being flipped? Because yea, if a HTTPS bit was flipped the whole packet dies. So is this issue blowing up all sorts of traffic everywhere?
Yes, all bits are being flipped. TLS connections drop because the message can't be authenticated, HTTP or other plaintext protocols will continue on with bad data.
So yes, it is blowing up all sorts of traffic everywhere. You just don't notice when it is plaintext.
Thanks for sharing this! I have long suspected that TLS does this. It is great: for many applications preventing bitflips in transit is arguably more important than privacy. Is there an authorative source where this is documented?
If you aren't interested in why this works just the Introduction to the RFC explains the intent, specifically what we care about here is that TLS delivers:
"Integrity: Data sent over the channel after establishment cannot be modified by attackers without detection."
And further notes "These properties should be true even in the face of an attacker who has complete control of the network".
I have ATT fiber in Texas and was having issues recently, probably for a few days, where DNS props would just fail(I use Google's DNS), huge pauses in page loads with the occasional just doesn't. Happened over the long weekend IIRC and was sporadic enough that I didn't look into it further. I thought it had largely cleared, but now I'm wondering about some ongoing page load pauses..
For me, TLS errors would happen every so often after we had used all our data allowance and our connection was being shaped to a slow speed. Once the speed returned to normal it was fine. I always thought it was the slow speed but maybe there were bugs in the shaping software.
I had AT&T via Sonic's FTTN in an effort to avoid Comcast/AT&T directly, especially since Sonic does not have data caps despite running on AT&T's network and actually respects their customer's privacy. Unfortunately my experience was so terrible and unreliable that I decided to give up and finally sign up with Comcast, and just have to mind that 1.2TB data cap.
The Bay Area of all places should not have such terrible ISP infrastructure, but here we are.
It's because AT&T does not follow net neutrality. It has fast lanes and slow lanes, which cause a lot of problems for work VPNs even when down the street. This TLS handshake bit flip is just yet another issue AT&T has, including the 1.1.1.1 DNS issues, and everything else.
I'm on Sonic.net fiber over AT&T too. To get the equivalent fiber from Comcast out here it's $270 a month. However, if you're lucky enough Sonic.net fiber that doesn't run over AT&T is amazing.
Huh, I had Sonic fiber over AT&T and loved it. In May I moved out of the bay area and now have Comcast cable. I'd do pretty much anything to get Sonic's level of quality out here.
I was seriously considering signing up for Sonic service, and I have Comcast now. Guess I won't do that. I hesitated because I really really hate AT&T. Guess I should have trusted that feeling.
It's shitty, but just FYI: you can pay Comcast an additional $30/mo for unlimited data. They hide the option, but it was ultimately the least bad option to handle my WFH traffic after Comcast re-instated the caps.
You can get Comcast Business at your residence. The overall bandwidth is lower, but you're priority traffic, and there are no caps. Plus, static IPv4 and IPv6 are available.
I had to switch also after missing days of work due to not being able to use the internet with AT&T. They just kept trying to send new routers, which was not the problem. And I thought ISPs in Alaska were bad, I never thought I'd have worse service in the Bay Area.
I have AT&T fiber. I noticed Wikipedia started to have trouble loading around then, assumed Wikipedia just had capacity issues. Script definitely shows corruption.
For this type of issue, I would recommend trying the nanog mailing list. You often see network admins ask for someone "with clue" at a different company when they get the runaround by tier 1/2 tech support.
This is absolutely the right way to "informally" escalate things to people that know what they're doing at big ISPs.
I'd recommend most newbies to the list show their work and post what's broken and how you know it isn't your fault. The investigative work on that Twitter thread is top notch and would do it in a second.
NANOG and outages@ are the two mailing lists that I've been subscribed to forever and are indispensable if you do operations.
"It depends" - Your NIC could be doing some tcp offload for you, at which point any checksum you'd observe with tcpdump/wireshark would be flat out wrong.
Generally - both the tcp checksum and the 802.11/Ethernet CRC isn't all that strong, so it's entirely within the realm of possibility of having bitflip patterns that fool both.
My mobile.twitter.com traceroute prefers going through that path, as does en.wikipedia.org (both of which have sucked for me) while a Google route (to 172.217.6.78) hops through 12.122.149.186.
373 comments
[ 4.1 ms ] story [ 252 ms ] threadI'm on the preorder list for Sonic, for the past.. almost 2 years now. In the meantime, I'm using resold ATT through Sonic. $10/mo more than ATT directly, but no MITM and no data caps.
Edit: Reading the tweet thread, classic interaction: ATT says "please click this to check your connection", guy replies back and says "I think we know more about networks that you do, please get a network admin in here".
I'm in the same boat. If the dozens of IT pros who are complaining about this can't get AT&T to swap out a single router card, what hope do most folks have?
If it happened to the rest of us at somewhere else, we are probably out of luck...
I can't even download the page 3 times in a row w/o corruption
If you use Twitch you'll randomly have the video break with a connection failure in the middle of stream.
Note: Only in certain areas. In SF they string their own fiber.
I might have to go back to the local cable company if this continues.
Anyone experiencing this in SoCal?
Cellular service isn't all that great either.
I suspect if the droughts didn't make people get rid of their green grassy lawns homeowners would be more amenable to seeing green network boxes every few houses. It looks awful in the context of concrete sidewalks, though.
1) Sprout the streetlamp from the box, add a car charging port.
2) Hire an artist to paint the box with artwork (property owner gets to pick from option set). Consider it to be city owned art installations.
https://www.dslreports.com/forum/r31593060-Equip-What-is-thi...
This map[0] shows the major faults, but each one of those is a lot of small minor faults that all could snap things and shift. One small fault right by where I grew up is about three blocks long. Last I checked had one earthquake to its name: a magnitude 5.0 aftershock of Loma Prieta
[0] https://usgs.maps.arcgis.com/apps/webappviewer/index.html?id...
SV is the perfect place to not bury infrastructure. There's no weather to worry about, and mostly people aren't going to shoot down the fiber to try to claim scrap metals.
You can bury conduits, by the way, and not cables or fiber directly. This allows you to avoid digging again just to install fiber, for example. There are ways to do it right. And having the infrastructure on poles is not a panacea either. New providers are not necessarily allowed to use the poles. The weather might not be crazy, but the poles are already overloaded and a little windstorm will disrupt electricity, coax cable, or fiber. And of course, those overloaded poles are crazy ugly.
When I used to work in the telecom industry, burying conduit or cable in the ground was anywhere from 3x (bury in some dirt on the side of a county highway) to 20x (directional drilling in a heavily populated city where there's utilities all over the place) the cost per foot compared to hanging it aerially from a labor standpoint.
However, as you correctly point out, there may be restrictions on what you can hang on the poles and where, and oftentimes you'll find poles where it turns out it never should have had the number of attachments it did, but guess who gets to foot a large part of that bill if they want on?
But even then, I've seen absurd lengths gone to in the name of not digging. On Martha's Vineyard, I believe they wound up using a super-special Self-Supporting fiber that could be hung in or near the Power area of a Pole. Yes, that requires a far more trained/well paid worker than normal aerial work. Also, in that region, NESC 250C/D comes into play which makes it even more of a PITA. But it still was far cheaper than putting cable in the ground.
I wonder whether Teraspan or other Vertical Directed Conduit would be a good fit for the bay area (Saw-cut a minimal depth in the street, just lay in a special zip-up conduit for fiber or twisted pair.) If the weather doesn't tend towards large temperature shifts it works well.
Speaking of which, a couple drawbacks worth noting for buried conduit; You have to go out and do your markings, or pay someone to do them for you when a dig request is made, and you have to be ready to handle the repair when someone inevitably forgets to call or the markings are done incorrectly.
I thought that was less of a "won't" and more of a "can't" between the lack of taller building towers and the radio interference zone for the airport means the directional antennas skip a slice there.
This will probably get fixed if 5G becomes a proper thing and there's a lot of micro-cells along the "bay area's biggest parking lot".
The people who would call their provider about bad cell coverage in their house are the same people that would go to city hall and demand that no cell towers be built in the city.
Makes a change from the city council members usual practice of denying Vallco permits and claiming Apple employees are hiring prostitutes and molesting high school students. (I did not make this up.)
https://twitter.com/HousingValley/status/1125959134329708545
https://twitter.com/HousingValley/status/1154781703262498816
Wonder if someone should take them up on it.
We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it.
That was in 2019.
Do they own their own fiber in some areas?
They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets.
I would probably expect this to be some network card or cable or connector is failing though...
Now, if you had written JavaScript to detect and remove these ads, and they went around that, then you might be able to construct a DMCA 1201 claim and sue the ISP for circumventing what is legally considered DRM. Yes, JavaScript can be legally protected DRM. The law doesn't say it has to be good DRM, it just has to have the effect of controlling access to a copyrighted work. And the safe harbors the DMCA provides ISPs wouldn't protect them in this case.
It's an interesting juxtaposition of how tolerant TCP/IP is of random bit flips, yet how intolerant application layer protocols are.
I would use traceroute to find a common bad point for everyone. It is also possible that the networking point where the problem occurs is invisible to traceroute as it could be part of a provider network probably MPLS but at least the common ends of the tunnel would be visible.
The fact that it is a a specific interval indicates a stuck bit in memory.
Some good previous public stories about such incidents https://www.verizondigitalmedia.com/blog/being-good-stewards... https://twitter.com/cperciva/status/1309568337408454658
How do you mean? I use traceroute from time to time but I’m not sure how it would apply in a case like this. Feel free to elaborate :)
Edit to make the comment more useful: If anyone is curious, look up "ECMP hashing." There are probably tons of parallel paths through AT&T's network, and to narrow down to the hardware causing problems, they will need to identify which specific path was chosen. Hardware switches packets out equally viable pathways by hashing some of the attributes of the packet. Hash output % number of pathways selects which pathway at every hop.
Hardware does this because everyone wants all packets involved in the same "flow" (all packets with the same src/dest IP and port and protocol (TCP)) to deterministically go through the same set of pipes to avoid packet re-ordering. If you randomly sprayed packets, the various buffer depths of routers (or even speed of light and slightly different length fibers along the way) could cause packets to swap ordering. While TCP "copes" with reordering, it doesn't like it and and older implementations slowed way down when it happened.
When implementing this functionality, the naive hardware designer will strip the existing CRC from the packet, modify the contents of the packet and then reuse the handy dandy CRC calculation block to place a newly calculated CRC on the packet. Similar choices are made for the adjustment of the IP/TCP/UDP checksums. If any errors are introduced in the contents of the packet by the data path prior to the new CRC is calculated, this results in the CRC being "corrected" to include the erroneous data.
A far more understanding hardware designer will instead calculate how to adjust the CRC by the changes introduced in the packet contents. Sadly, this is far more complicated to get right, and it goes against the drive of hardware designers to reuse blocks of code wherever possible. Every hardware designer working on networking has a block of Verilog or VHDL code to calculate and append a CRC to a packet. Only the most dedicated will attempt to apply only the delta needed to the CRC or checksum.
For people like me who aren't smart enough to figure it on their own, this stackexchange answer seem to explain how it's done: https://cs.stackexchange.com/questions/92279/can-one-quickly...
On the application side the effects were quite bad, as the data was mainly XML and, depending on where the bit was flipped, it could impact the data or the XML structure. The data had its own CRC/hash, so the packets were cleanly rejected by the application. Unfortunately the XML library from the message queue engine and the ESB we were using did not like at all when the bit flipping occurred in the XML tags (it seems fuzzing tests were not done at that point) so the message processing got stuck and we kept getting bad messages in the queues. Even worse, the queues could not be cleaned with the normal procedures because the application wanted to first display info about the messages inside - and that failed.
The network debug was non-trivial because of that header consistency - the network devices did not report any kind of packet issues, so we had to sniff the different network segments to identify the culprit. From the application point of view, we had to delete the whole message queue storage to get rid of the bad messages, and let the application handle the rest (luckily it was designed with eventual consistency and self-healing).
There was an escape hatch, but the conditions to hit it were a bit complex. Implementing new message filtering of this kind at 2AM while the system was down was not feasible.
I really wish we'd just have municipal ISPs here. I'm only a few miles from Google HQ, but the internet options are crap.
Years ago I had gotten an "ATT fiber coming soon" notice, but still not here. In the end its simply the usual issue with internet access in the US -- ISPs have zero incentive to invest/improve, there's no meaningful competition. With everyone WFH its even worse -- oversubscribed networks are causing trouble all over (esp if you use a cable ISP).
When a city (or someone like Google) brings fiber in a meaningful way, it only then lights a fire under the incumbents' ass. In theory cities with non-buried lines (like Mountain View) may be easier to upgrade; but it still depends on ATT granting pole access (which they promptly slowwalk).
For me, the problem has mostly manifested as web pages failing to load or appearing to be loading forever. Generally when I refresh the page would load quickly as expected.
https://twitter.com/bmastenbrook/status/1335400747794530304
It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS handshake is an error and the connection dies.
Tech support, even when Twitter accounts with a lot of followers message them, is completely useless, they just say that they don't show any outages at your location. They need to be flooded with complaints for someone to look at this, or maybe someone from AT&T is on here that can get it looked at...
They are on Facebook and Linkedin, you have seem them.
Edit: Actually I take that back, I have seen one of the issues where sometimes web sites will not load at random, and a reload fixes the issue.
The files differ after anywhere from 1sec to 5sec for me, and it's always the same character positions, and it always seems to be the same lines, and the same number of lines.
The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking.
But if you're one of the 5% you're fucked.
If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.
The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensated so that he actually cares about solving the problem) and the charge is refunded only if the problem indeed ends up being on the ISP's side. This should self-regulate as anyone wasting the engineers' time for a simple problem they could resolve themselves would pay for that time.
About 20 years ago I got escalated to high-tier Comcast support for an issue that turned out to be a little of A, a little of B: Comcast (Might have been @Home, based on the timing) required that your MAC address be whitelisted as part of their onboarding process. Early home routers had a "MAC Address Clone" feature for precisely this reason. At some point, the leased network card got returned to Comcast. Our router continued to work just fine... until about a year later, when the local office put that network card into some other customer's home. We started getting random disconnects in the middle of the day, and it took forever to diagnose, as the other customer was not particularly active with their internet use. Whose fault was it? Ours, for ARP Spoofing? Theirs, for requiring the spoofing?
That makes sense.
When the cable modem issued a DHCP request, the CMTS would have been configured to insert some additional information (a "circuit-id") into the DHCP request as it relayed it to the DHCP server.
The short version is that the "competent tech" looked at the logs from the DHCP server, which would have showed that the "same" cable modem (i.e., MAC address) was physically connected to either 1) two different CMTS boxes or 2) two different interfaces of the same CMTS.
How would one cable modem be physically present in two different locations at the same time? Obviously, it wouldn't.
At that point, either 1) there are two cable modems with the same burned in address or 2) one of the two cable modems is cloning/spoofing its MAC address. Which one of those is more likely?
(If you're interested in the details, try "DHCP Option 82" as your search term.)
For instance: "This might just be a loose cable, can you unplug each end of it, and plug it back in?" invariably elicited an "I've already done that", or a brief pause followed by "okay, there, I did it, do you see it?". Lies, often.
But: "Alright, I want you to try turning the cable around for me, yeah swap it end for end. Sometimes you get a connector that just doesn't fit quite right, but it works the other way around and it's faster than sending a tech to replace it", would often get a startled "Oh! I hadn't thought of that, one moment..." and then the customer actually DOES unplug the thing, and what do you know, they click it in properly this time.
Then one day, she tells me that she tells people to unplug the power cord, cup the end in their hand, and then plug it back in.
Suddenly, I really liked her. It's a genius move that makes them think they've done something obscure, but she really just wanted them to actually check the cable.
Everything they do is driven by metrics, and their contracts are written around KPIs like maintaining a ludicrously short average time to answer, short handle times, and open/closed ticket ratios. If they do not hit these metrics, then the outsourcing company owes service credits. The incentives do not align with making customers happy and solving their problems. Everything is geared towards deflecting users with self-help options, simple scripts for the agents, and walking the line of hitting those metrics with the fewest number of warm bodies.
It's a pretty hellish business.
For the last several years, I've gotten my mobile service through an MVNO named Ting. On the rare occasion that I've needed to call support, there's no IVR, just a human who typically answers on the second ring. They speak native English, and have never failed to solve my problem either immediately or with one prompt escalation.
They're so jarringly competent I wonder how they still exist, if being obnoxiously incompetent is apparently a business requirement.
Luckily, I already knew how to fix them. I found the job to be a cake walk and quite liked helping people. But I had to listen to people around me fumble through it.
It was frustrating at the time, but my favorite thing that happened was that I was admonished twice for having average call times that were too low. To them, that's a warning that someone is just getting people off the line without fixing their problems.
They monitored calls and they said I'd never received a complaint, but the system would keep flagging me for low call times so I had to artificially raise them. They suggested that I have a conversation with the clients.
I didn't, and I didn't stay there much longer, but it was quite a crazy situation. But I also felt much less pressure to handle calls quickly after that, too, which was nice.
I used to play counter strike/Starcraft in my middle school years. I pretty much figured out I had consistent packet loss with a simple ping test. I was on the phone with Time Warner every other day for months. They kept sending the regular technicians, at one point ripping out all of my cable wires in the house to see if it fixed the problem. Nothing worked, I kept calling, at this point I had the direct number to level 3 support. They saw the packet loss too. Finally, after two or three months they send out a Chief Engineer. Guy says I’ll look at the ‘box’ on one of the cable poles down the block. He confirmed something was wrong at that source for the whole area. Then it finally got fixed.
Took forever dealing with level 1 support, and lots of karenesque ‘can I talk to your supervisor please’, but that’s literally what it took.
So yeah, if you want stuff like this fixed, stay professional, never ever curse, consistently ask to speak to the supervisor, keep records, and keep calling.
Small shout out to the old http://www.dslreports.com/ for being a great support community during the early days of broadband for consumer activism in terms of making sure you got legit good broadband.
I ended up just cancelling the service and signing up for one of their competitors instead.
Anyway I was told on more than one occasion by different telcos that the standard operating procedure for many techs was to take the call, do nothing, and call back 20 minutes later and ask if it looked better because ... often enough it did.
It always was.
When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path worked.
Someone who has taken the time to report an issue is probably one of hundreds or thousands who had an issue and didn't think it could be fixed and shrugged it off. We certainly can't fix everything, but weird network shit like this can be fixed, and it's worth escalating, because when you get it fixed, you can also figure out (hopefully) how to monitor for it, so it doesn't happen again.
OTOH, I didn't work for the phone company. We don't care, we don't have to, we're the phone company. https://vimeo.com/355556831 (sorry about the quality, I guess internet video was pretty lowdef in the 70s :P)
The most memorable one:
Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they answered, the party on the other end didn't seem to hear them. They seemed to be talking to _someone_, but not the party they were connected to. Eventually they hung up. Sometimes, a customer would place a call, and be on the other end of the same situation -- whoever answered would say hello, but the two parties didn't seem to be talking to each other. Off into the void. They'd try again, and it would work, usually, but repeats weren't uncommon.
So everyone's looking at system logs and status alarms and stuff, and what else changed? There were two new racks of echo-cancellers placed in service last night, could that cause this? Not by any obvious means, I mean e-cans are symmetrical and they were all tested ahead of time. There was a fiber cut out by the railroad but everything switched over to the protect side of the ring OK, didn't it? Let's check on that. Everyone's checking into whatever hunch they can synthesize, and turning up bupkus.
Finally around lunchtime, one of the techs bursts into the ops center, going "TIM! I GOT ONE I GOT ONE IT'S HAPPENING TO ME, PATH ME! okay look I don't know if you can hear me, but please don't hang up, I work for the phone company and we've got a problem with the network and I need you to stick on the line for a few minutes while we diagnose this. I know I'm not who you expected to be talking to, and if you're saying anything right now, someone else might be hearing it, but that's why this is so weird and why it's so important YEAH IT CAME INTO MY PERSONAL LINE and that's why it's so important that you don't hang up okay? I really appreciate it, just hang out for a few, we'll get this figured out..."
Office chairs whiz up to terminals and in moments, they've looked up his DN and resolved it to a call path display, including all the ephemera that would be forgotten when the call disconnects. Sure enough, it's going over one of the new e-cans. Okay, that's a smoking gun!
So they place the whole set of new equipment, two whole racks of 672 channels each, out-of-service. What happens when you do that is the calls-in-process remain up, but new calls aren't established across the OOS element. Then you watch as those standing calls run their course and disconnect, and finally when the count is zero, you can work on it. (If you're doing work during the overnight maintenance window, you're allowed to forcibly terminate calls that don't wrap up after a few minutes, but that's verboten for daytime work. A single long ragchew is the bane of many a network tech!) The second rack was empty of calls in _seconds_, and everyone quickly pieced together what that implied -- every single call that had been thus routed was one of these problem calls where people hang up very quickly. This thing had been frustrating hundreds of callers a minute, all morning.
With the focus thus narrowed, the investigation proceeded furiously. Finally someone pulls up the individual crossconnects in the DACS (a sort of automated patch panel, not entirely unlike VLANs) where the switch itself is connected to the echo-cancellation equipment. And there it is. (It's been too long since I spoke TL1 so I won't attempt to fake a message here, but it goes something like this:) Circuit 1-1 transmit is connected to circuit 29-1 receive, 29-1 transmit isn't connected to anything at all. 1-2 transmit to 29-2 receive, 29-2 transmit to 1-1 receive. Alright, we've got our lopsided connection, and we can fix it, but how did it happen in the...
> including all the ephemera that would be forgotten when the call disconnects
Interesting to know there is information which is not logged. I’m guessing keeping this info, even for a day, would have helped isolate the issue?
How did the echo cancellers pass testing?
The funny thing is that if everyone played along they could have had a mean game of telephone going.
All that was true, the failure happened when it was being taken out of testing config and into operational config. Either nobody considered that that portion could fail, or the urgency to add capacity to a suddenly-overloaded network meant that some corners were cut. (Marketing moves faster than purchasing-engineering-installation...)
Oh, and as to the point about keeping the call path ephemera. Yeah probably, but in a server context, that'd be akin to logging the loader and MMU details of how every process executable is loaded and mapped. Sure, it might help you narrow down a failing SDRAM chip, but the other 99.99999% of the time when that's not the problem, it's just an extra deluge of data.
He loved all of that stuff, absolutely hated when everything went to computers. Quit and became a maintenance man at a nursing home, commercial laundry repair guy then finally retired this year in his late 70’s (due to Covid) after working maintenance at a local jail.
I believe the #5 ESS machine itself is always in closed cabinets, so it's likely that what you're remembering was the toll/transport equipment, or ancillary frames. Gray 23-inch racks as far as the eye can see!
Depending on how old that part of the office was, they were likely either 14' or 11'6" tall with rolling ladders in the aisles, or 7' tall and the only place they'd have laddertrack was in front of the main distributing frame.
As for magnetic core, if you could see it mounted in a frame, what you probably saw was a remreed switching grid, which is a sort of magnetic core with reed-relay contacts at each bit, so writing a bit pattern into it establishes a connection path through a crosspoint matrix. It's not used as storage but as a switching peripheral that requires no power to hold up its connections. (Contrast with crossbar, which relaxes as soon as the solenoids de-energize.)
Remreed was used in the #1 ESS (and the #1A, I believe), and is extensively documented in BSTJ volume 55: https://archive.org/details/bstj-archives?&and[]=year%3A%221...
https://upload.wikimedia.org/wikipedia/commons/7/7e/1A_Frame...
I just remember thinking it looked awkward getting to the equipment under them.
I don’t know if the ‘5E’ as he called it was actually in operation yet, he ended up moving us all out of state to take a job developing and delivering training material for it...I think that’s what finally broke him lol. Hands on kinda dude.
I’ll have to hit him up later today to see if he remembers ‘remreed’ (he will). Thanks for the info!
At night, traffic was often low enough that you could hear individual call setups and teardowns, each a cascade of relay actuations rippling from one part of the floor to another. The junctor relays in particular were oddly hefty and made a solid clack, twice per call setup if I recall correctly, once to prove the path by swinging it over it to a test point of some sort, and then again to actually connect it through. On rare occasion, you'd hear a triple-clack as the first path tested bad, an alternate was set up and tested good, and then connected through.
Moments after such a triple-clack, one of the teleprinters would spring to life, spitting out a trouble ticket indicating the failed circuit.
The #5, on the other hand, was completely electronic, time-division switching in the core. The only clicks were the individual line relays responsible for ringing and talk battery, and these were almost silent in comparison. You couldn't learn anything about the health of the machine by just standing in the middle of it and listening, and anyone in possession of a relay contact burnishing tool will tell you in no uncertain terms, that the #5 has no soul.
https://www.telcomhistory.org/connections-museum-seattle/
Have you heard any of Evan Doorbell's telephone tapes[1]? It's a series of recordings mostly from the 1970s, but with much more recent narration, exploring and sort-of documenting the various phone systems from the outside in. Might be interesting to see what they figured out, and what they didn't :)
[1] http://www.evan-doorbell.com/
It had happened before and then magically fixed itself a few days later.
One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basically cabling guys, better for finding a bad cable than debugging a packet loss. The problem for me was that my ipv4 static ip addresses would not receive traffic.
I was at wit's end after a week and I debugged the thing myself. By looking at EVERY bit of data on the router, I found mention of the blocked packets in the firewall log. I would clear all the logs, and found even with the firewall DISABLED, the firewall log would see and block incoming packets I was sending using my neighbor's comcast connection.
I called AT&T, but this time mentioning "firewall is completely off, but packets are blocked by the router and showing up in the log" was concrete enough for them to look up a (known) solution.
The fix was to disable the firewall, but to enable stealth mode. wtf?
To be clear, this was a firmware bug, and caused dozens of calls to AT&T, lots of heartache and finger pointing always in my direction.
I should also mention at the start of this fiasco, I checked the system log and noticed they pushed a firmware update to the modem at the time the problem started. Strangely after one call to the agent, that specific line disappeared out of the log file, but other log entries remained. hmmm.
since then, they basically screw up my modem every month or two - they push new firmware and new "features" appear (like the one that sniffs and categorizes application traffic like "youtube" and "github"). It also helpfully turns wifi BACK ON when I had disabled it. I immediately go turn if back off and then they immediately send me a big warning email that my DSL settings have been changed.
This is all because AT&T believes that the edge of their network is not the PON ONT/OLT, but rather, the router they issue you. If you want to be on their network, you have to use their router as some part of the chain.
My latest discovery is that in doing this, the router can actually get super hot operating at gigabit speeds for extended periods of time. When this happens, it magically starts dropping packets. Solution? Aim a fan at the router so it has "thermal management."
Total. Garbage. I'd switch to Spectrum if they had decent upload speed, but alas, they don't in my building.
[0]: https://particle17.xyz/posts/amplifi-alien-thoughts/#appendi...
[1]: https://www.att.com/support/article/u-verse-high-speed-inter...
Another option is getting the 802.1x certificate out of a hacked router, but it's not possible as far as I know on the 5268ac. You could buy a hackable ATT router but they're not cheap. Some sellers even sell the key by itself.
Mysteriously, doing this fixed an issue I previously had where SSHing into AWS would fail.
[1] https://github.com/jaysoffian/eap_proxy
https://github.com/MonkWho/pfatt
I'm currently using eap_proxy with my BGW210, and it's been a huge improvement, but I fear the day the device needs to be replaced with a newer model.
However, it has 5Gbit Ethernet, hasn't re-enabled WiFi on automatic firmware updates, and has only screwed with my IP Passthrough configs once which was resolved with a router reboot. (that was possibly my router's fault, it seemed like it was unable to fetch a new DHCP lease)
... only if you want people to actually be able to escalate. Suspect AT&T are going to lose a lot less money over this issue than hiring an extra high quality support person would cost
All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc.
CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.
https://xkcd.com/806/
I've had this exact experience, except that it wasn't a dream.
Back in 2010 I had a weird issue where my cable connection would sometimes completely block the connectoin right after the DHCP response (we had dynamic IPs back then). This would go on for a couple of hours, until the IP lease expired, then my connection would come back. Luckily, I was running an OpenBSD box as my router which allowed me to diagnose the problem. But it was also impossible to explain to the servicedesk employees.
One evening it happened again, and I called the servicedesk, totally prepared to do the 'yes I have turned it off and on again' dance. But to my surprise the employee that I got on the phone was very knowledgable and even said that it was very cool that I had an OpenBSD box as a router. He very quickly diagnosed that someone in my neighbourhood was 'hammering' the DHCP service by not releasing his lease (a common trick to keep your IP address somewhat static). This caused a double IP on the subnet, and the L2 switch to block traffic to my port.
He asked me "do you know how to spoof your MAC with an OpenBSD box?". Then I knew this guy was legit. He instructed me to replace the last 2 bytes of the MAC with AB:BA (named after the music group). They had a separate DHCP pool for MAC addresses in that range. If they ever saw an ABBA mac address on their network, they knew it was someone who had connectivity issues before.
The problem was immediately solved, and I had a rock-solid internet connection for years, with a static IP!
I ended up chatting about networking and OpenBSD a bit, before I (as humble as I could) told the guy I was a bit flabbergasted that someone as knowledgable as him was working on the servicedesk.
It turned out, he was the chief of network operations at the ISP (the biggest ISP in my country). He was just manning the phone while some of his colleagues from the servicedesk were having diner.
Sometimes miracles do happen.
For evidence, see: “Retention departments give you the best rates if you threaten to cancel” (which then caused companies to have to rename the retention departments and change the policies.
Nice find! I had been wondering why I had been seeing odd TLS failure messages recently.
Only in certain areas! Within San Francisco on overhead-cabled blocks (for example) their lines are their own.
Has anyone claimed to see these flipped bits on a domain other than example.com?
See e.g. https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d...
Given that (it seems) only AT&T customers are complaining, and (it seems) it only affects servers on EdgeCast.
It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packets.
There is no reason for it to touch the TCP header.
And yet, an unfortunate number of L2 switches do exactly that. :(
https://en.wikipedia.org/wiki/Carrier-grade_NAT
Can you name any? Just curious
The problem is routers slow down the high speed serial signals from fiber to by splitting the bits over a large number of slower speed signals internally. Often those wider busses are a multiple of 16 bits. For example, one ASIC I know of moves things around in 204 byte chunks. (Might have been 208, been a while.) Anyway, the problem is that if there is a defect in one of those parallel elements it will always flip bits in the same offset position mod 204 bytes, which is the same position mod 16 bits. If the hardware is degraded enough, it can end up flipping two bits in the same position, and that has a fairly good chance of passing the checksum.
Ethernet has proper CRCs on packets, which is a lot less vulnerable to shenanigans like this, but unfortunately those can end up being checked on the way in, discarded, and then re-generated on the way out of a router. If anything is corrupted in the middle of the switch ASIC, nothing notices and it passes along. I once helped troubleshoot an issue in our network where a BGP packet was corrupted in this way. The flipped bits ended up causing a more specific route to be generated, and we had the world weirdest BGP route hijack within the bounds of our own data center.
My ATT modem (Arris bgw-210-700) may have gotten a firmware upgrade recently as I found some settings I shut off got re-enabled. I use it for VDSL but the same model family is used for ATT fiber.
0 and 0 are still 0 and 0,
1 and 1 are still 1 and 1,
but 0 and 1 become 1 and 1 (or 0&0)
and 1 and 0 become 1 and 1 (or 0&0).
I remember seeing this wackiness when I bridged two address lines on an EEPROM with the tiniest amount of solder.
RoHS/tin whiskers strikes again?
So either the bit corruption is such that is not detected by the checksum
Or ATT is doing something nefarious and touches layer 4 and corrupts the data while doing so
https://www.evanjones.ca/tcp-checksums.html
With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.
Routers don’t touch the TCP header at all
But in any case it's irrelevant in this context as ATT shouldn't be doing any NAT
Another quality product by Apple ;P
https://twitter.com/KielO/status/1335640544496689152
That tweet in particular doesn't show any retransmits.
tcpdump/wireshark gets a little hard to read at times; especially when the packet dump is a lie: all those packets marked red for bad checksums are from the dumping machine, and the checksums are wrong because the NIC is filling them in, and the capture interface doesn't get to see what they are). Perhaps the other people in the thread who said mac os was ignoring bad checksums were also confused; or perhaps it does ignore bad checksums, it's pretty bad at networking (it can't handle a synflood in 2020 because it's got synhandling code from 2000)
I wouldn't have expected Apple to purposefully break the checksum, just like they I don't think they purposefully have no synflood protection because they pulled the TCP/IP stack at the turn of the century and never pulled it to get the many many many upgrades from upstream (although, they did add on MP-TCP, so there's that). I wouldn't be surprised if tcp checksums had stopped working ages ago, possibly because of a aggressive driver, and nobody noticed. Kind of like how if you spawn a few thousand threads that just sit around sleeping, it will delay watchdog kicks and the kernel will panic. (also, from reports on here, not personal experience)
Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and your crappy Internet mangling the data in transit, in either case the TLS protocol design says to "alert" bad_record_mac which your browser or similar software will probably treat as a failed connection, even if it happens mid-way through an HTTP transaction.
Because TLS guarantees integrity even if your fiber is a complete shit show, any TLSPlaintext records which do get from one end to the other are guaranteed to be as intended.
So yes, it is blowing up all sorts of traffic everywhere. You just don't notice when it is plaintext.
https://tools.ietf.org/html/rfc8446
If you aren't interested in why this works just the Introduction to the RFC explains the intent, specifically what we care about here is that TLS delivers:
"Integrity: Data sent over the channel after establishment cannot be modified by attackers without detection."
And further notes "These properties should be true even in the face of an attacker who has complete control of the network".
What's a "DNS prop"?
Also found a Reddit thread (https://www.reddit.com/r/bayarea/comments/k87h2f/psa_att_fib...) about the issue that advises to contact the executive service customer support (https://investors.att.com/resources/contacts#customer-servic...) if your calls to regular customer support don't fix the issue - I did, we'll see how it goes
The Bay Area of all places should not have such terrible ISP infrastructure, but here we are.
I'm on Sonic.net fiber over AT&T too. To get the equivalent fiber from Comcast out here it's $270 a month. However, if you're lucky enough Sonic.net fiber that doesn't run over AT&T is amazing.
If you must Comcast, at least make sure you Comcast Business so that you get connected to non-idiot support people.
Script definitely shows some issues.
https://archive.nanog.org/list/join
https://seclists.org/nanog/2019/Aug/103
I'd recommend most newbies to the list show their work and post what's broken and how you know it isn't your fault. The investigative work on that Twitter thread is top notch and would do it in a second.
NANOG and outages@ are the two mailing lists that I've been subscribed to forever and are indispensable if you do operations.
edit: apparently osx doesnt drop these packets according to the twitter thread.
Generally - both the tcp checksum and the 802.11/Ethernet CRC isn't all that strong, so it's entirely within the realm of possibility of having bitflip patterns that fool both.
Almost certainly 12.242.117.22
My mobile.twitter.com traceroute prefers going through that path, as does en.wikipedia.org (both of which have sucked for me) while a Google route (to 172.217.6.78) hops through 12.122.149.186.
It’s not impossible that people from AT&T read messages posted there.
You people are amazing!