HN Explorer

Substantial ATT Fiber Issues in SF Bay Area?

16 points by rconti ↗ HN
For the past 24 hours, my fiber connection in Redwood City has been virtually unusable. Gigabit one moment, 0Mbps upload the next.

I’m finding substantial anecdotal evidence of ATT Fiber home internet issues from San Jose to San Francisco on the Sonic forums (for those of us using resold ATT fiber), Nextdoor, heck even AT&T’s own website problem map shows outages all up the Peninsula, but then says my zip (94063) has no problems.

Is this getting coverage anywhere I can follow? Does anybody know what’s actually going on? Feels like substantial congestion breaking connections. Typical web page load takes easily 10 tries before it successfully completes.

21 comments

[ 2.3 ms ] story [ 50.4 ms ] thread
backprop1989
I’m nearby near Woodside. Same thing for me. Interestingly enough my issue has been upload speeds near 0.01 Mbs, downloads are generally fine.

Rumor has it the lines were cut somewhere but I haven’t been able to confirm.

rconti
Same symptom. Upload starts off fast and tails off to 0.01.

I’m surprised I can’t find a test suite I can run on my system that will hit, say, 50 various websites and report on what’s happening- slow elements, failures on certain types of elements, etc. With the bad upload I suspect many requests on a web page load simply fail to arrive at the remote, but I really don’t know.

toast0
I've got a test suite [1] for another issue that might be somewhat helpful. It does probing for path MTU issues; server -> client probing may all happen on a single connection if everything goes well, but client -> server probing is one connection per probe, so that might be enough to trigger some issues; although the probe size search uses some shortcuts to reduce the number of probes.

Anyway, if you see anything unusual in the testing, maybe we can get some pcaps and see if anything stands out. Of course, getting that information up the chain to someone at ATT who can fix it, would be up to you :)

(email in profile if you see anything useful)

[1] http://pmtud.enslaves.us/

puntium
seeing the same thing here. Seems to have started sometime on Friday. Were you able to get any confirmation?
rconti
Nope. Nothing good. Sonic forums reporting problems back to last Monday. No idea what’s going on.
puntium
Things seem back to normal for me now.
teaearlgraycold
In Redwood City on Sonic. fast.com is showing peaks of 5 Gb/s and sustained 2.7 Gb/s down and 1.5 Gb/s up. No issues in the last few days at all. So whatever the issue is it's very localized.
rconti
You’re on native sonic, though, not ATT fiber, correct?
ev1
AFAIK 10G is only Sonic native
[deleted]
(comment deleted)
[deleted]
(comment deleted)
pwncake
Likely ATT being resold as Sonic in that area
syntaxing
Hah reminds me of an issue with AT&T in the Bay Area in 2020 during peak Covid. Somehow someone with a strong background in this narrowed it to a bad RAM where it’s flipping one of the bits which breaks SSL (?) on one of the stations within the tracenet. Everyone tried complaining on twitter to AT&T since telling customer rep did not help at all (went right over their head). This issue persisted for a couple weeks but I moved before they fixed it.
rconti
THANK YOU!! This issue was in the back of my head. Someone even created a utility where you could curl and it would tell you what % of the time the header was getting flipped. I’ll search it out later.

This subconscious memory caused me to look at SSL vs HTTP. It seems to be problematic on HTTP too. But oddly it’s not yet been an issue with curl http. But http in a browser is troublesome and inconsistent.

EDIT: found it.

https://news.ycombinator.com/item?id=25335936

jklm
For anyone still running into this - using a VPN seems to fix things but not sure why.

Some kind of throttling / connectivity issues on ATT's side?

toast0
A lot of issues like this end up being tied to one or two specific routers; in a previous issue, it was found to be ram with a stuck bit in a particular line card connecting ATT and a major CDN (or somebody) in Palo Alto [1]. Especially if it's an issue on a connection to CDNs, your VPN connection is likely to traverse a different path, and the path from the VPN to the destination will also likely traverse a different path.

Most of the time, networking issues are at a single link on the path, and forward and reverse paths are different too, so it can be hard to track down. BGP does a great job of quickly routing around totally broken links, but there's not usually any sort of quality or utilization metric included, so a link where there's lots of problems but BGP still works is going to stay up and continue to be utilized.

[1] See discussion here, but I didn't see the specific resolution https://news.ycombinator.com/item?id=25335936

jklm
That makes sense, thanks for explaining it. Got on the phone with support and they've been receptive and are escalating it.
rconti
I have no direct info, but they've updated their outage maps to at least show the issue, and I've heard from numerous sources that they're actually working on something, so that's a plus.
puntium
confirming that using tailscale and an exit node in the google cloud seems to fix the problem for now
tlefkowicz
My Sonic resold AT&T fiber internet has been down for 8 days. For 8 days, Sonic has said that AT&T won’t dispatch a technician while there is an outage in the area. I’ve been a consistent squeaky wheel so Sonic has gotten AT&T to dispatch a repair technician tomorrow morning. We shall see if the tech shows up (typically 50-50), and if they do, if they bring the right equipment and can actually fix our connection. It has been a maddening week+ in SF.