We're experiencing a network outage. Stay tuned at https://www.fastmailstatus.com . We're in close communication with our provider trying to restore service.
Diminishing returns. This is the first time I’ve ever heard of a FastMail outage, so it seems their architecture is commercially reasonable as it relates to outages.
> Weâre experiencing network issues with several providers. Get updates on fastmailstatus.com as the situation progresses.
(including the quirks characters, copied directly).
That implies the outage isn't universal, and some ISPs/locations may not be affected. Is anybody able to access through a particular VPN (or country)? If so, posting that here could help others get access to their inboxes faster.
Thanks for posting this. I live in China and thought Fastmail was finally being blocked by the GFW. Over the past year I've had intermittent issues with connectivity and I've wondered if FM's time in China was up. Are there any HN readers in China that have experienced similar issues with FM?
Core router issue at NYI - apparently it's not just us, though we're one of their major non-US customer-base customers, so it affects a lot of our users. I spoke to their techs briefly to confirm details, but otherwise am leaving them alone to do their jobs!
It's particularly annoying that it's a partial outage and our main 66.111.4.x range is the most affected (I suspect because it runs through DDoS protection that our more hidden ranges don't) so most our monitoring is running just fine, but customers can't get in. And it would take an hour (thanks "sensible" DNS timeouts) to move the traffic with the knobs we have access to.
Good news! Mail access has been restored for users on some major networks. We apologize to those customers who still cannot reach their Inboxes - we're working to restore full access as fast as we can. No mail has been lost during this downtime. https://www.fastmailstatus.com
16 comments
[ 3.4 ms ] story [ 47.7 ms ] threadtraceroute to mail.fastmail.com (66.111.4.148), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 2.582 ms 4.760 ms 4.748 ms
2 142.254.XXX.XXX (142.254.XXX.XXX) 14.748 ms 41.677 ms 43.837 ms
3 agg60.vnnzca2401h.socal.rr.com (76.167.27.73) 44.784 ms 47.498 ms 47.498 ms
4 agg11.vnnycajz01r.socal.rr.com (72.129.14.96) 47.489 ms 47.479 ms 47.499 ms
5 agg29.lsancarc01r.socal.rr.com (72.129.13.0) 47.475 ms 47.466 ms 47.452 ms
6 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 47.458 ms 20.444 ms 19.780 ms
7 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 24.220 ms 25.283 ms 25.654 ms
8 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 27.438 ms 29.922 ms 29.924 ms
9 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 32.877 ms 33.666 ms 34.208 ms
10 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 35.399 ms 35.901 ms 36.878 ms
11 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 34.177 ms 34.492 ms 34.462 ms
12 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 31.085 ms 28.907 ms 30.610 ms
13 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 27.499 ms 35.354 ms 36.167 ms
14 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 38.251 ms 39.675 ms 39.303 ms
15 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 36.657 ms 36.652 ms 38.186 ms
16 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 39.633 ms 46.706 ms 45.702 ms
17 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 43.710 ms 34.548 ms 42.508 ms
18 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 42.399 ms 38.360 ms 39.058 ms
19 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 41.678 ms 43.000 ms 44.069 ms
20 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 44.913 ms 45.806 ms 46.209 ms
21 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.609 ms 50.484 ms 51.050 ms
22 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.087 ms 55.045 ms 39.316 ms
23 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 52.467 ms 52.470 ms 52.450 ms
24 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.596 ms 43.254 ms 46.939 ms
25 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.514 ms 53.067 ms 54.051 ms
26 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 61.721 ms 61.724 ms 61.715 ms
27 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 54.020 ms 54.011 ms 54.003 ms
28 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 51.628 ms 53.483 ms 53.482 ms
29 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 56.560 ms 57.956 ms 58.392 ms
30 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 57.149 ms 58.921 ms 57.447 ms
We're experiencing a network outage. Stay tuned at https://www.fastmailstatus.com . We're in close communication with our provider trying to restore service.
1:34 AM - 1 Aug 2018
https://twitter.com/FastMail/status/1024573967393157121
> Weâre experiencing network issues with several providers. Get updates on fastmailstatus.com as the situation progresses.
(including the quirks characters, copied directly).
That implies the outage isn't universal, and some ISPs/locations may not be affected. Is anybody able to access through a particular VPN (or country)? If so, posting that here could help others get access to their inboxes faster.
Since the MX are spread between NYI/AS11403 and AS201628, I think that all mail delivery should still be fine.
I'm receiving occasional push notifications on my iOS FastMail app, can't actually read them though!
It's particularly annoying that it's a partial outage and our main 66.111.4.x range is the most affected (I suspect because it runs through DDoS protection that our more hidden ranges don't) so most our monitoring is running just fine, but customers can't get in. And it would take an hour (thanks "sensible" DNS timeouts) to move the traffic with the knobs we have access to.
Good news! Mail access has been restored for users on some major networks. We apologize to those customers who still cannot reach their Inboxes - we're working to restore full access as fast as we can. No mail has been lost during this downtime. https://www.fastmailstatus.com
3:09 AM - 1 Aug 2018
https://twitter.com/FastMail/status/1024598033793601536
No data lost, and the servers themselves have been happily humming along the entire time, so we don't even have filesystem checks to worry about.
Congrats and thanks, good luck with the search for another upstream or two :)