Now think about if some gmail accounts got Thanos snapped out of existence. What kind of digital death would that be as you would NEVER be able to recover your accounts, at least for most people who don't have 2FA.
> Now think about if some gmail accounts got Thanos snapped out of existence. What kind of digital death would that be as you would NEVER be able to recover your accounts, at least for most people who don't have 2FA.
An interesting problem - I have a gmail address, and also one on my personal domain (which uses g-suite for email). The personal domain's backup is a gmail address, and the gmail's backup is my personal domain. When I set that up ages ago, I genuinely never thought about what happens if gmail itself implodes.
I guess I'm setting up a... icloud??? backup email for a bunch of stuff shortly.
Can they call someone when there is an issue with gmail? Is there a contract that give them some leverage and guaranties? What is the process in the case of a missing business-critical email being lost and needing recovery?
For business critical operation, leave such answers unanswered can be pretty risky.
Now would be an excellent time to ask people who use gmail for business critical operation how well that SLA and telephone support work in practice.
I know for example companies that use email for handling customer orders at their stores. I can just imagine the loss if the system was down for hours a few days before Christmas, or worse, actually lost the data.
The funny thing is part of that situation started exactly 24h ago as I too saw increased bounces from AWS SES towards Gmail addresses. Not for everyone though, to be fair.
If anyone from AWS SES is reading - please do not deliver bounce receipts for gmail for the time being - it makes everyones situation much worse.
"Type: Permanent; SubType: General; Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser y128si147264pfg.177 - gsmtp"
This is pretty much the worst response possible. Hard bounces mean that email delivery services are going to start automatically removing, or at least stopping delivery to, entire slews of email addresses.
A lot of clean up is going to be needed as a result of this.
To add some more details, when using a 3rd party email delivery service, those services will either black-list or just outright remove email addresses when they get a hard bounce "email address no longer exists" message back.
Some providers make re-adding an address after a hard bounce a non-trivial task, since after all, the authority on that email address just said it doesn't exist.
Yes. I email hundreds of thousands of Gmail users each week (yes, double opt in, they all want the mails!) and we immediately delete any user for whom any Gmail error comes up at all in order to keep a solid delivery record with them. Sounds like we might have deleted 80% of our list if we'd sent today..!
Yes, we're unusual in not relying on third parties for list management. We can rollback. Or I might just comment out the 'unsub on hard bounce' code for the rest of the week..! :)
Logically you'd expect unsubscribe to only act after lots of bounces of this format when the address has been receiving mail fine before. It also seems reasonable not to trust such bounces for the entire domain for a while when this happens to lots of other addresses that have worked fine before. Not that I expect software currently works this way, but it does seem like a common sense thing to code in.
I mean, it's possible, but you'd need to queue up a day's worth of bounces, do the analysis, and then handle the bounces asynchronously later on to do that.
Most systems operate more immediately in isolation on individual addresses than that right now, because such analysis is generally not needed (until today, of course ;-)).
Mail agents already queue emails that bounce though; it's a matter of changing the conditions for when you retry and/or unsubscribe. I imagine you can do the analysis in real time too... just look at the bounce and see if it pertains to an email you sent to in the past, and if so, increment some rolling counter for that domain.
Yes, most likely! That is a common approach for 'soft bounces' in most list management systems (e.g. MailChimp).
The problem here is Gmail has been throwing out "NoSuchUser" errors which are an instant unsub in most systems because Gmail takes repeated delivery to non-existing addresses into account for deliverability purposes.
I'm extremely paranoid about email hygiene, tiny bounce rates and high delivery rates, so we aggressively unsubscribe troublesome addresses (often to the point of getting reader complaints about it) for many reasons beyond that, however.
That better describes what I was trying to say, yes. Reputation then affecting deliverability.
Over 80% of our subscribers use Gmail so to say I'm paranoid about maintaining a good record with them is an understatement ;-) Gmail is a huge weak link for us.
So new think to do: Quarantine addresses instead of deleting them and if for one provider most addresses fail don't give them another (maybe manually triggered) try later one.
(And if no such thing is detected deleted quarantined mail addresses.)
My guess is that how most email service providers handle this - they don't actually delete the email and just have a flag on it - bounced, complain, unsub. This way the list owner can run an export and see all the status code.
I really cannot believe they did not immediately hack in a new rule to their SMTP server: never return a 5xx (permanent failure), instead return a 421 (temporary failure try again later).
That simple fix buys them 24-72 hours to solve this properly.
Yeah, it burdens servers sending mail to them because now they have to hold on to all mail (including mail that really is permanently undeliverable) for another day or so, but that's still better than what's happening right now.
5xx error results in suppression list addition of an email, so future emails won't be delivered (by most ESPs), and not returning MX response would probably be just as bad, or worse (or result in millions/billions of emails being re-queued due to timeouts?)
His solution would result in exponential retry failures baked into most services, which would buy them a few hours, and result in no lost emails, and no suppression list additions.
This outage seems to have lasted for about 2.5 hours. Probably this was fixed by rolling back whatever caused it. (I don't think the rollout was finished before they resolved it; my mail server sends a lot of emails to Gmail addresses, and even at peak I was only seeing maybe about 1/3 mails be rejected.)
There is no way that putting in a hardcored hack like that would have been faster. Making the change is, of course, fast.
But then you need to review it (and this is a super risky change, so the review can't be rubber stamped). Build a production build and run all your qualification tests. (Hope you found all the tests that depend on permanent errors being signalled properly). And then roll it out globally, which again is a risky operation, but with the additional problem that rolling restarts simply can't be done faster than a certain speed since you can only restart so many processes at once while still continuing to serve traffic.
The kind of thing you describe simply can't be done by changing the SMTP server, in 2.5 hours. The best you could get is if there was some kind of abuse or security related articulation point in the system, with fast pushes as required by the problem domain but still with the sufficient power to either prevent the requests from reaching the SMTP server at all, or intercept and change the response.
As a trivial example, something like blocking the SMTP port with a firewall rule could have been viable. Though it has the cost of degrading performance for everyone rather than just the affected requests.
My mail server logs show about 20 failures in all of the last week until yesterday 20:43 CET, then 350 failures between 20:43-00:21, then nothing after that. So fair enough, from the client side rather than the status page it looks like 3.5 hours rather than 2.5.
But still, given that resolution time, the suggested solution of changing the SMTP server is absolutely ludicrous.
Mailgun send a warning mail about increased bounces from our account. Sure, they know what's going on... but we send 4-5 digit mails per hour - it's a lot of bounces
That means I can't just resend the the emails blindly, because I'm too scared to trigger some sort of automatic suspension...
(I don't do this regularly, so I'm not familiar with all features... additional mail verification could help probably ....)
They should be returning 421 for backend outages so that sending servers queue and retry the emails. 550 can be interpreted by some as deleted [1] or even banned accounts in some cases. Maybe someone here could convince them to change the logic that occurs during an outage.
Yah. Maybe there's an unexpected way that things can fail resulting in 550's. But maybe at Google's scale you should have some kind of kill switch to stop answering SMTP or to not send permanent errors at all, so that you could flip a switch and prevent the worst consequences of this rather than let it go on for a couple of hours.
A lot of people will lose transactional email messages, because of this.
I'd absolutely hate to be hit by this at this time. Thankfully I've made an time investment to run my own mail server years ago. A handful of times it broke down, it either went offline or started returning 4xx codes due to misconfigured or broken milter after an update. Neither meant lost messages from normal senders that use queuing MTAs.
Same for me, mainly for privacy concerns. And I back it up daily to my local NAS. It's so easy to configure and run your own mail server, that I'm surprised we are the minority in the tech community.
> It's so easy to configure and run your own mail server
Is it? Is dealing with IP reputation, getting your emails accepted by major providers, and being on the hook for fixing everything yourself very easy? I haven't tried, so I don't have personal experience, but I've heard enough horror stories to think that it's not a good use of my time.
I had an IP reputation issue and managed to resolve it after some time.
TLDR: Before you spin up a mail server, check if your IP address is on any of the blacklists [0]-[1] as well as Proof Point's list [2]. If it is, then try and get a different IP address.
I spun up a hosted server on Digital Ocean and received an IP address. I checked several black lists from a few email testing/troubleshooting sites [0] and [1] and all was groovy; my IP address wasn't on any list.
I got a bunch of 521 bounces when I tried emailing a neighbor who had an att.net address.
So, I checked the troubleshooting websites, and my IP address was listed as clean.
My logs said I should forward the error to abuse_rbl@abuse-att.net, so I did.
Those emails were never delivered, because abuse-att.net had its own blacklist. I was getting 553 errors. In the logs, the message from their server told me to check https://ipcheck.proofpoint.com.
Proof point runs their own blacklist that some enterprises use (e.g. att and apple [3]). I checked their list, and lo and behold, my IP address from Digital Ocean was blocked [2]. Digital Ocean wasn't able to remove the IP address from their blocklist and suggested I spin up a new droplet with a different IP address.
I didn't want to do that, so I sent Proof Point an email that went unanswered; the email asked them to remove my IP address. I forgot about the issue for five or six months (this is a personal server), and ran into the issue again a few months ago. So I sent Proof Point an email again, this time with different wording emphasizing that "my clients" were having delivery issues. Within a day, they removed my IP address from their block list.
So, my main suggestion is to check if your IP address is on any of the blacklists as well as Proof Point's list before you start on your server. If it is, then try and get a different IP address.
Does anyone have more "enterprise" lists, like Proof Point, to check?
Mine are accepted by Gmail so I am good. Considering how dominant Gmail is, that's all that really matters.
Regarding getting a bad IP rating, normally that's due to having an insecure config, like acting as an open relay, or not having DKIM enabled. There are lots of tutorials online about this, if you know Linux it really is easy.
Sending side of the MTA can be set up manually in about an hour on a Debian server, with dmarc, dkim, spf, etc. Make that a day if you want to read up on and understand each of the things in more detail, if you haven't configured them before. There's really not much to play with in this direction for a typical personal mail server.
Receiving side is where there is a great range of options, and many things to try and have fun with. You can have anything from a single catchall mailbox with no filtering, no GUI, and a simple IMAP or POP3 access for MUA, to a multi-account, multi-domain setup with server side filtering, database driven mailbox and alias management, proper TLS, web MUA access, etc. It can also be built up gradually, starting from very simple setup to something more complicated so that you never lose account of how things work.
I also had the same hard bounce (when emailing from a non-gmail address -- fastmail -- to a gmail address). Sent it again minutes later and then it worked.
Yeah, I get that. But why punish the Desktop users with mobile interface? We used to punish Mobile users with Desktop interface back then, we just reversed the problem, didn't actually solve it.
They should be 2 separate things. 2 separate css files with @media select. 2 separate button sizes and styles.
Yeah, I like rich, data-heavy, information dense desktop displays. Mobile UIs are such garbage, not to mention shit like reddit that only is able to process a request 30% of the time.
Doing all the work twice is expensive. It’s not just the actual work of doing the design, but then you’re 2x testing costs too (unless you YOLO one of the modalities). More cost effective and better quality result to make a choice to make the platforms more uniform rather than trying to optimize for both. This obviously only holds when there’s a massive disparity in revenue. If your PC/laptop users make up a non-trivial parts of revenue it can start to make sense (but again, generally only if doing so will help you retain that market or get better revenues). The other thing unifying things does is it helps engineering and design teams by removing complexity.
All of the above is general trends of how these decisions are made. There will always be counter examples or situations those aren’t good ideas (or that someone has made a mistake applying a lesson to the wrong situation).
Saying an entire group of people is lazy or dumb is not a particularly insightful way of looking at anything that helps your understanding of the situation or learning what kind of results different incentives yield.
> Saying an entire group of people is lazy or dumb is not a particularly insightful way of looking at anything that helps your understanding of the situation or learning what kind of results different incentives yield.
Maybe, but when you're talking about the armies of designers at the multi-billion-dollar tech company Google not going through the effort to maintain two stylesheets, I think "laziness" is an accurate description.
I get a completely different site on my phone than on a desktop/laptop.
In fact, they maintain far more than two designs - in addition to two native apps, there's a mobile website, the desktop website, and the basic HTML version. On top of that, they have multiple display density options for desktop (which admittedly is mostly just adjusting padding), redesigned the desktop site a few months ago, and had Inbox for a few years. On top of that, you can (some of these without a reload) change whether there are separate inboxes for various labels, add/remove a reading pane, and split threads into individual emails.
I don't think Google is lacking in potential to maintain a website.
Is the plain HTML version under even the most basic maintenance? I was under the impression that was the old interface and they just kept it around because people liked it/slower countries.
The mobile vs desktop versions you posted are likely the same codebase with minimal (if any) differences. My understanding is that generally such things are accomplished transparently with flex layouts that automatically adjust to screen size
> Is the plain HTML version under even the most basic maintenance?
I doubt much work is being done on it, but presumably they at least make sure it works; I mentioned it because a few posts up (edit: you) mention testing (rather than initial design) as the reason why having multiple designs is so difficult.
> The mobile vs desktop versions you posted are likely the same codebase with minimal (if any) differences....
It's entirely possible that they're derived from a similar codebase at some point, but what reaches the browser is significantly different - it's barely responsive, based on user-agent, and appears to be significantly different obfuscated blobs of HTML, CSS, and JS.
I can’t speak to it but just because something is tested occasionally doesn’t mean the testing budgets are the same or serve the same purpose.
For example, the feature set required to support the HTML page could be frozen and the APIs backing them stable with no need to change. So testing isn’t really necessary. Alternatively, there’s just API changes being made to remove dependencies on deprecated code and so the testing coverage comes from the testing that happens of that API surface through other means. Finally, it could be that the HTML page is even fully staffed to support emerging markets. That’s a different budget potentially than the budget for the “rich” UI.
Again, my point isn’t to argue over the specific business pressures and practices Google has for their email UI. This requires a level of knowledge I don’t think either of us possess. All I’m trying to do is illustrate that there could be all kinds of pressures why the system is the way it is, but dismissing it as “laziness” or “stupidness” on the part of the designer is itself a lazy and stupid conclusion to make without concrete evidence. I generally assume that’s not the case and look for the incentives/pressures those people are under until there’s overwhelming evidence those people are actually stupid/incompetent (and even then, the question becomes what structures, incentives, pressures were in place to put those people in positions they shouldn’t occupy).
No, designers did not get lazy. Devices got weirder.
It used to be that you could check the width and height of the viewport and say something like “320px wide? Must be a touch interface, deploy the big buttons”. Then tablets got big and it was like “1024px? Could be a laptop, but it’s probably an iPad, which has a touch interface, deploy the big buttons”. Then laptops got touch screens, then the Surface Studio came in and was like “HAHAHAHA”.
Now the game is “1920x1080? Could be a big tablet with touch, or a 1080p monitor without touch, or a non-maximized window on a Surface Studio with touch, or maybe it’s a monitor without touch hooked up to a laptop with a touchscreen and our window could get moved between them at any time...”
Nowadays, there’s no single reliable way to tell if a page is going to have to support touch until it gets a touch event, by which point you’ve already rendered the UI and it’s too late.
Some websites probably do have settings around this, but that gets to a point someone else mentioned: you would have to basically design, build, test and maintain two UIs. Except now with the kicker that one of those layouts is only used by the 5% of your userbase that both knows that the option is available and chooses to take you up on it.
You don’t have to ask the user, there is a media rule for querying whether the device is currently using coarse or fine pointer input[0] (though, of course, it relies on the OS not lying, which is not a given).
That's the "entropy only goes up" or "all available space will be filled with complexity" rule of software orgs. Employees are paid to add new features, regardless of how useless they are. The only situation when they stop adding features is when hardware or something else doesn't allow it. Otherwise we'd see 1GB webapps. But when they reach this boundary, they start redoing existing stuff, because otherwise they'll get fired for inactivity. This kind of bs is difficult to stop even when you're paying their salaries and monitor the results. For example, a marketing person would keep adding useless bloat to justify his salary (he really needs his paycheck!) or a programmer would keep refactoring some bs to satisfy his purism (funded by your money, of course). Just think about it: if a competent programmer approaches you and explains how the product is mostly finished and the remaining microimprovements won't add any value to your business, would you continue paying him for doing nothing?
Over the past 24 hours, I've had GitHub request that I re-verify my gmail three times (roughly 22 hours ago, 2 hours ago, and now), each time resetting my primary email's status to "Undeliverable" and "Unverified"
The triggering event may be an email bounce. I get a lot of github notifications sent to my email, and the failure of just one/a few may trigger the reverification.
When that happened I panicked a little, realizing how much Google Sheets data I had that wasn't really backed up anywhere since Sheets files in Google Drive are basically just links. I started a Takeout, but it looks like I wasn't the only one - it took well over a day to complete.
Be sure to verify that it worked. Some settings of Takeout don’t download docs/sheets/slides files. I don’t remember what the default is, unfortunately.
It can as well be Google, just the paid Apps version. Zero time to get used to a different UI. I suspect there must be a solution to easily migrate all your tags and filtering rules. (Tags are the killer feature to me. Outlook sort of has them but they are less flexible.)
Having run my own mail server for over a decade, I have yet to have a single time where the server responds by Permanent error of accounts not existing and with email bouncing.
Losing incoming email is pretty much the worst case scenario when it come to configuration errors. It about as bad as not having backups, in that both cases results in unrecoverable loss of data.
I see two problems here: The likelihood your service is restored before you spin up your own mail server, and the fact that, not expecting this failure, their DNS may have a fairly lengthy TTL.
Keep in mind other stuff like DNS will go down randomly. At least they won't result in a permanent address-doesn't-exist error, but you'll be putting out potentially more fires that way.
You can forward handling to a provider, like gmail. The idea is that you own your email address and can switch providers more easily if you are not satisfied with them or they turn out to be evil.
As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days.
Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.
Yes, I’ve been hearing about this issue from non-technical friends too. An explanation of “X crashed” helps even if they don’t actually understand what X is. The fact that someone figured it out and knows is reassuring.
That varies based on location and regional dialect. Here in the northeast US, I remember using singular they/their since the 80's. It would be interesting to know when this become popular elsewhere.
80s in Australia too, been hearing/using it my whole life.
Though with respect to 'ages' apparently it's been around since at least the 14th century but certain purists tried to stamp it out at various times (just like the singular 'you' which no one currently has grammatical issues with I hope).
The same reason it ever mattered how you refer to people, politeness and respect. If someone you consider "him" asks you to refer to them as "her" it's like someone asking you to call them by their full name "Rebecca" instead of "Becky" or "Jonathan" instead of "Jon". If you like and respect them, you do as they request because things which matter to them matter to you, and being polite to them is important to you. If you ignore what they ask, call them what you want, you communicate that you don't respect them and don't want to be polite, that you want to dominate and 'win' instead.
> "Pronouns can mean whatever you want them to mean"
Only one way. A specific person asking you to use a specific pronoun for themselves is wildly different from you unilaterally and universally saying that all women should feel included by the word "him" because "him" has no meaning anymore.
'Singular "their" etc., was an accepted part of the English language before the 18th-century grammarians started making arbitrary judgements as to what is "good English" and "bad English", based on a kind of pseudo-"logic" deduced from the Latin language, that has nothing whatever to do with English... And even after the old-line grammarians put it under their ban, this anathematized singular "their" construction never stopped being used by English-speakers, both orally and by serious literary writers.'
It's about not being a fucking asshole to people because you want to stand on [your own stupid] principle.
If you make an honest mistake and somebody attacks/humiliates/whatevers you for it, they're the asshole. If you can't manage to have some bare minimum consideration for somebody who is their own person just as much as you are, you are the asshole. I see both of the above happening pretty regularly these days.
This stuff is a real gimme, so really the most charitable interpretation of your comment is that you are being disingenuous for edginess' sake.
There used to be times when people didn't care for technicalities like this because the focus was on the person's contribution to the discussion.
Now that everyone's replaceable, the popular culture desperately tries to shift focus into arguing about pronouns and terms.
Watch out, this is a road to nowhere. Forcing others to use the right pronoun won't build up your retirement fund, but will distract you from worrying about not having one. And the fact that you care about it more than about your opponent's T-shirt color could be an indication that you are being manipulated to not think about the long-term things.
I’ve read a lot of horseshit reasons against showing the least possible shred of humanity by using someone’s correct pronouns, but “then the Illuminati will steal your 401k!” is a new one.
This is where it crosses from insightful into conspiracy theory territory for me. People seem perfectly capable of groupthink-deluding themselves. Why cheapen your argument by postulating some master manipulator when it's not necessary for the deeper point you're making?
It will only lead to people focussing the discussion to challenge this particular aspect, or them disregarding all you've said, instead of engaging with the actual meat of the argument.
I remember some people tried to get BLM into German discussions, which made absolutely zero sense, as we have a complete different history and culture. Now I see this popping up. I really hope Europe can get some cultural distance between itself and the USA in the near future. The time is ripe.
In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.
Well, google did use a bunch of off the shelf technologies in the early days, but now it is obvious that there is no vendor on earth that could supply the infrastructure to run Gmail.
I thought SMTP was specifically designed for this (with support for multiple MX entries, queuing at the sender MTA side, etc.) and there's an easy hard boundary at the user mailbox level you can use to partition your system.
It should not be a problem that gmail is "down". Unless this would be happening for more than a few days, noone would lose e-mail. It's a problem that it's not returning a temporary error code, but permanent one.
It is pretty clear that accepting a TCP connection and reading the bytes of the email from the sender is not the problem. Google is bouncing messages with an error like "that user doesn't exist". This would lead one to believe that some instances are having trouble looking up users, and that doesn't scale super easily. If the product guarantees that it will reject invalid email addresses (which is nice of them, not required by any spec), there has to be a globally consistent record of what email addresses are valid, and the accepting server has to look it up and react in the time that the sender is still connected to the mail server. You can't queue those and send the bounce later (there is no reliable "from" field in email; the only way to correctly bounce is while the sender is still connected). This basically means that you have on the order of milliseconds to accept or reject the email, so merely starting up a another replica of your SMTP daemon isn't going to mitigate this issue. The chokepoint is querying the list of users to see if you should bounce or accept the email. They made it hard on themselves by providing messages like "that user doesn't exist", but... it is nice when you email someone and you get the message "they got fired, sorry" instead of silence. So they made their system more complicated than it needed to be, for a better user experience, and now they are fighting a breakage.
I doubt that the delivery stack would 550 for mere trouble looking up an account. This smells more like the identity system was incorrectly returning authoritative denials.
Yeah, that sounds right to me. I would expect to see a temporary rejection with DEADLINE_EXCEEDED or something like that.
I think a lot of time and effort is spent categorizing errors from external systems into transient or permanent, and it's always kind of a one-off thing because some of them depend on the specifics of the calling application. It definitely takes some iteration to get it perfect, and it's very possible to make mistakes.
If it really doesn't want to accept emails for addresses that it doesn't know are valid, a well-behaving email server should send temporary failure codes when it can't look up if addresses are valid, and let the sender retry later when the address lookup is working and it can give a definite acceptance or rejection of the email. This is not even remotely a new problem, it comes up in email systems all the time because even at much smaller than Google scale they tend to be distributed systems. Someone screwed up.
> This basically means that you have on the order of milliseconds to accept or reject the email, so merely starting up a another replica of your SMTP daemon isn't going to mitigate this issue. The chokepoint is querying the list of users to see if you should bounce or accept the email.
You don't have milliseconds. You can take quite some time to handle the client. 10s of seconds for sure. For example default timeout for postfix smtp client when waiting for HELO is 5minutes.
You don’t really have to speculate, they disclosed yesterday that yesterday’s issue had to do with the automated quota system deciding the auth system had zero quota:
Maybe they have world class DevOps, but they also have way more things that can go wrong than the vast majority of businesses. It's kind of remarkable that the entire world can be pinging Google services and they have ~99.9% uptime.
Googler but nowhere near Gmail, so just educated speculation:
* We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. (e.g. what if you're in a situation where rolling back could make the problem worse? we might be Google, but we don't have magic wands)
* Debugging new failure modes is a coin flip: maybe your existing tools are sufficient to understand what's happening, but if they're not, getting that visibility can in itself be difficult. And just like everyone else, this can become a trial and error process: we find a plausible root cause, design and execute a mitigation based on that understanding, and then get more information that makes very clear that our hypothesis was incomplete (in the worst case, blatantly wrong).
We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation.
As Douglas Adams says, "The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair."
Rollback proof bugs are rare, but boy howdy are they exciting. I think I've only seen one so far (unless you count bad data / bad state that persists after a bad change is rolled back... which can also be pretty exciting)
You can build rollbacks out of rollforwards, although it certainly isn't particularly fun. You patch an update to version N version code so that it's higher than N+1 and roll out the N+2 labelled N.
I speculate that for many companies, work from home has been at most, less impacting than they thought.
However, I'd speculate that in this instance, when you get that .0001% problem, less hands on deck makes work from home aspects less easier. Akin to remotely fixing somebodies PC over standing behind them.
With that premise I'd speculate in this instance that whilst not the root cause, may of been a small ripple that led to that root cause and/or lead to a slower resolution than what would normally get.
Those speculations aside, it will only highlight what that some tooling needs to adjust for remote workers as does design and set-ups more. Water cooler talk is not just for gossip and a counter would be more regular on-line group socialising at a work level so that not only the companies but the workers can fully adapt and embrace the work medium; But so the kinks and areas that need polishing can be polished and made better for all.
Lastly, I'd speculate that I'm totally wrong and yet what I said may well anecdote with some out there and resonate with others.
You might be right for the smaller company where physical access to the machines in the data center is necessary at a certain point in the troubleshooting process. I work at such a place myself. I would guess, however, that Google moved beyond that quite some time ago. It's simply not practical, with or without having offices with people in them.
Agreed. And I'd hope that their plan B of "get the whole team on Hangouts" isn't met with connection / auth issues. Kinda feel bad for the googlers. Hope they get this right.
> It blows my mind that Google can even have problems like this.
When you operate at Google's scale then everything that can go wrong, will go wrong. Google does an amazing job providing high-availability services to billions of users, but doing so is a constant learning process; they are constantly blazing new trails for which there are no established best practices, and so there will always be unforeseen issues.
I didn't know being a programming contest script kiddie and grinding leetcode all day automagically makes you a world class Devops, just cz you got into Google.
Yes, apps are highly distributed. Yes, roll-outs are staggered and controlled.
But some things are necessarily global. Things like your Google account are global (what went down the other day). Of course you can (and Google does) design such a system such that it's distributed and tolerant of any given piece failing. But it's still one system. And so, if something goes wrong in a new and exciting way... It might just happen to hit the service globally.
When things go down, it's because something weird happened. You don't hear about all the times the regular process prevented downtime... because things don't go down.
If there is something I've learned from AWS outages (they tend to publish detailed post-mortem), no matter how you design your architecture in a distribute way you will always have Single Point of Failure (SPOF) and sometimes discover SPOF you didn't think of.
Sometimes it's a script responsible of deployment that will propagate an issue to the whole system. Sometimes it's the routing that will go wrong (for example when AWS routed all production traffic to the test cluster instead of production cluster).
I wonder if Gmail is just not a very well maintained codebase. Here's an issue where old emails just become inaccessible. Not fixed for over a year and they've locked the thread so I'm starting to wonder if they actually deleted the emails by mistake.
Some of the internal formatting that Gmail uses has changed over the years, so more likely than not the API that parses the stored message for display in the Gmail UI is just throwing some kind of error.
Right but the version of an email message you download via IMAP is different than the version of an email message you see in the Gmail UI. That's my point, that the error is probably in the way Google is processing messages for Gmail, so you wouldn't see it in IMAP or via the API.
Uneducated speculation, some sort of security incident. Whenever there is a major security issue in the wild, one of the big providers tends to have a problem within a few days.
I don’t work at Google, I’m at a different big tech that’s in the news frequently. Sharing inside info on an ongoing incident is a great way to get fired. Big tech companies are way different than startups where everyone can do a bit of anything. There are people whose job it is to handle that communication. You make their job a lot harder if you disclose information. The company is so big that as an engineer you may not know all the factors involved in what would hurt the company long term - undisclosed relevant litigation, compliance commitments, partner obligations, etc.
How much do you hate it as an engineer when sales people make tech promises to customers without asking you? For comms people, engineers leaking info publicly feels the same way.
What you're saying makes sense but I don't think it really applies to anything the OP said. The "non-confidential" qualifier indicates to me that they only want people to share what they can responsibly.
And the parent post’s point is that there are people whose job it is to specifically share that information, and so we should let them do their job. They are the domain expert in this particular task.
I am very pleased to see this response, genuinely. Our Technical Curiosity aside, there are literally people and teams in such big firms dedicated for this.
Yes, I hate that but they do it nonetheless so I don't see why we have to be such good citizens suddenly. Google fucked up. Nice. It's 100% garantueed something stupid and technically interesting. Nothing to be sued over.
Which timezone is my local timezone? A bunch of geolocation systems think I'm in Paris. I'm in Melbourne. (My ISP bought an IP block that used to be registered to a French company).
If this had happened on self hosted email server, people would be claiming "this is why you don't self-host email". My company self hosts email and we have never had an outage of this sort. ever. All support emails to gmail accounts are now bouncing for us. It's also been more than 2 hours since this problem started.
The name for this is "victim blaming." Someone suffers from a catastrophe (or from abuse, but that's not what happened here), and you imagine a way that it could've been avoided, never mind that things could've gone just as badly wrong the other way around.
But far more consistently. Most users have no idea how the Internet works, let alone hosting something as complex as email. They'd just stop using email and switch entirely to Facebook.
is there a product you could use, like a cache layer for gmail? An investor I know would love that product. I understand things go down, but it's actually quite a pain to lose access to your existing emails and calendar events. Could he just use a Mail app like Outlook? Yes. But that's too complicated to set up I think.
Yes, they Internet has felt for lack of better word “weird” for the past couple days. Nothing specific but similar to your experience with random things not loading but refresh and it’s fine etc. Or strangely slow response on websites.
I reeeeally don't want to go down this rathole but a few days ago when my kids were having problems connecting to YouTube on both wifi and LTE I had begun to think that Trump pulled the 1. Stop Internet, 2. Attempt Coup thing.
While, unlike many other people, I do _not_ believe Trump is an idiot, or even dumb at all, I do not believe he has any knowledge of how to stop the internet even if he wanted to.
Plus, how would he send tweets if no internet? I think he likes being able to use Twitter more than he likes being President.
My cursory understanding of the whole "internet going dark" conspiracy is that this would not be used by Trump, but be used by the shadow government in co-operation with "Big Tech" in order to create panic and confusion amongst the masses so they can't organise, feel helpless and ultimately ask for "big government" and "big tech" to step in and save them.
For whatever reason, gp seems to be under the impression that it's Trump who would use this to his own advantage, which is not consistent with the prevailing conspiracy theories.
The "internet" deserves these outages to make people - and CEOs, CIOs, etc - realize that in-house ~~engineering~~ sysadmins used to exist for good reasons.
458 comments
[ 3.6 ms ] story [ 293 ms ] threadEdit: oops thought it was a text post
An interesting problem - I have a gmail address, and also one on my personal domain (which uses g-suite for email). The personal domain's backup is a gmail address, and the gmail's backup is my personal domain. When I set that up ages ago, I genuinely never thought about what happens if gmail itself implodes.
I guess I'm setting up a... icloud??? backup email for a bunch of stuff shortly.
Thankfully, we have backups, but we will have to move them to the inbox or elsewhere to have them processed.
Edit: As an update, we usually have at least 100 emails come in every hour, and I am seeing none since 4:02 pm EST
Not a good week for Google
For business critical operation, leave such answers unanswered can be pretty risky.
I know for example companies that use email for handling customer orders at their stores. I can just imagine the loss if the system was down for hours a few days before Christmas, or worse, actually lost the data.
Yes.
Other people’s automation scares me. I’m sure mine scares other people as well.
If anyone from AWS SES is reading - please do not deliver bounce receipts for gmail for the time being - it makes everyones situation much worse.
A lot of clean up is going to be needed as a result of this.
To add some more details, when using a 3rd party email delivery service, those services will either black-list or just outright remove email addresses when they get a hard bounce "email address no longer exists" message back.
Some providers make re-adding an address after a hard bounce a non-trivial task, since after all, the authority on that email address just said it doesn't exist.
This is going to be really ugly.
Kind of happy I had to do something else and I didn't burn hours investigating.
Most systems operate more immediately in isolation on individual addresses than that right now, because such analysis is generally not needed (until today, of course ;-)).
The problem here is Gmail has been throwing out "NoSuchUser" errors which are an instant unsub in most systems because Gmail takes repeated delivery to non-existing addresses into account for deliverability purposes.
I'm extremely paranoid about email hygiene, tiny bounce rates and high delivery rates, so we aggressively unsubscribe troublesome addresses (often to the point of getting reader complaints about it) for many reasons beyond that, however.
I think you mean "reputation purposes"?
If so, wow, that sucks. Their opaque rules have conditioned their counterparties to punish Google as hard as possible for a screwup.
Good for karma, bad for everyone though.
That better describes what I was trying to say, yes. Reputation then affecting deliverability.
Over 80% of our subscribers use Gmail so to say I'm paranoid about maintaining a good record with them is an understatement ;-) Gmail is a huge weak link for us.
"Gmail going down" would not have caused this problem. Even if all their SMTP servers went offline.
(And if no such thing is detected deleted quarantined mail addresses.)
That simple fix buys them 24-72 hours to solve this properly.
Yeah, it burdens servers sending mail to them because now they have to hold on to all mail (including mail that really is permanently undeliverable) for another day or so, but that's still better than what's happening right now.
His solution would result in exponential retry failures baked into most services, which would buy them a few hours, and result in no lost emails, and no suppression list additions.
That is a better scenario, than 5xx.
There is no way that putting in a hardcored hack like that would have been faster. Making the change is, of course, fast.
But then you need to review it (and this is a super risky change, so the review can't be rubber stamped). Build a production build and run all your qualification tests. (Hope you found all the tests that depend on permanent errors being signalled properly). And then roll it out globally, which again is a risky operation, but with the additional problem that rolling restarts simply can't be done faster than a certain speed since you can only restart so many processes at once while still continuing to serve traffic.
The kind of thing you describe simply can't be done by changing the SMTP server, in 2.5 hours. The best you could get is if there was some kind of abuse or security related articulation point in the system, with fast pushes as required by the problem domain but still with the sufficient power to either prevent the requests from reaching the SMTP server at all, or intercept and change the response.
As a trivial example, something like blocking the SMTP port with a firewall rule could have been viable. Though it has the cost of degrading performance for everyone rather than just the affected requests.
My mail server logs show about 20 failures in all of the last week until yesterday 20:43 CET, then 350 failures between 20:43-00:21, then nothing after that. So fair enough, from the client side rather than the status page it looks like 3.5 hours rather than 2.5.
But still, given that resolution time, the suggested solution of changing the SMTP server is absolutely ludicrous.
That means I can't just resend the the emails blindly, because I'm too scared to trigger some sort of automatic suspension...
(I don't do this regularly, so I'm not familiar with all features... additional mail verification could help probably ....)
[1] - https://en.wikipedia.org/wiki/List_of_SMTP_server_return_cod...
I am astonished that either (a) this switch has not been flipped yet or (b) this switch does not exist.
Somebody is incompetent here.
https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...
On the other hand, their status dashboard reported similar issues yesterday and here we are again: https://www.google.com/appsstatus#hl=en&v=status
I'd absolutely hate to be hit by this at this time. Thankfully I've made an time investment to run my own mail server years ago. A handful of times it broke down, it either went offline or started returning 4xx codes due to misconfigured or broken milter after an update. Neither meant lost messages from normal senders that use queuing MTAs.
Is it? Is dealing with IP reputation, getting your emails accepted by major providers, and being on the hook for fixing everything yourself very easy? I haven't tried, so I don't have personal experience, but I've heard enough horror stories to think that it's not a good use of my time.
TLDR: Before you spin up a mail server, check if your IP address is on any of the blacklists [0]-[1] as well as Proof Point's list [2]. If it is, then try and get a different IP address.
I spun up a hosted server on Digital Ocean and received an IP address. I checked several black lists from a few email testing/troubleshooting sites [0] and [1] and all was groovy; my IP address wasn't on any list.
I got a bunch of 521 bounces when I tried emailing a neighbor who had an att.net address.
So, I checked the troubleshooting websites, and my IP address was listed as clean.
My logs said I should forward the error to abuse_rbl@abuse-att.net, so I did.
Those emails were never delivered, because abuse-att.net had its own blacklist. I was getting 553 errors. In the logs, the message from their server told me to check https://ipcheck.proofpoint.com.
Proof point runs their own blacklist that some enterprises use (e.g. att and apple [3]). I checked their list, and lo and behold, my IP address from Digital Ocean was blocked [2]. Digital Ocean wasn't able to remove the IP address from their blocklist and suggested I spin up a new droplet with a different IP address.
I didn't want to do that, so I sent Proof Point an email that went unanswered; the email asked them to remove my IP address. I forgot about the issue for five or six months (this is a personal server), and ran into the issue again a few months ago. So I sent Proof Point an email again, this time with different wording emphasizing that "my clients" were having delivery issues. Within a day, they removed my IP address from their block list.
So, my main suggestion is to check if your IP address is on any of the blacklists as well as Proof Point's list before you start on your server. If it is, then try and get a different IP address.
Does anyone have more "enterprise" lists, like Proof Point, to check?
[0]: https://www.mail-tester.com/
[1]: https://mxtoolbox.com/blacklists.aspx
[2]: https://ipcheck.proofpoint.com
[3]: https://www.reddit.com/r/email/comments/6toxzr/ip_blocked_by...
Regarding getting a bad IP rating, normally that's due to having an insecure config, like acting as an open relay, or not having DKIM enabled. There are lots of tutorials online about this, if you know Linux it really is easy.
Receiving side is where there is a great range of options, and many things to try and have fun with. You can have anything from a single catchall mailbox with no filtering, no GUI, and a simple IMAP or POP3 access for MUA, to a multi-account, multi-domain setup with server side filtering, database driven mailbox and alias management, proper TLS, web MUA access, etc. It can also be built up gradually, starting from very simple setup to something more complicated so that you never lose account of how things work.
They should be 2 separate things. 2 separate css files with @media select. 2 separate button sizes and styles.
I know what happened - designers got lazy.
All of the above is general trends of how these decisions are made. There will always be counter examples or situations those aren’t good ideas (or that someone has made a mistake applying a lesson to the wrong situation).
Saying an entire group of people is lazy or dumb is not a particularly insightful way of looking at anything that helps your understanding of the situation or learning what kind of results different incentives yield.
Maybe, but when you're talking about the armies of designers at the multi-billion-dollar tech company Google not going through the effort to maintain two stylesheets, I think "laziness" is an accurate description.
Sure, but they already do all the work twice.
I get a completely different site on my phone than on a desktop/laptop.
In fact, they maintain far more than two designs - in addition to two native apps, there's a mobile website, the desktop website, and the basic HTML version. On top of that, they have multiple display density options for desktop (which admittedly is mostly just adjusting padding), redesigned the desktop site a few months ago, and had Inbox for a few years. On top of that, you can (some of these without a reload) change whether there are separate inboxes for various labels, add/remove a reading pane, and split threads into individual emails.
I don't think Google is lacking in potential to maintain a website.
Desktop (plain HTML): https://mail.google.com/mail/u/0/h/
Mobile: https://mail.google.com/mail/mu/mp/
The mobile vs desktop versions you posted are likely the same codebase with minimal (if any) differences. My understanding is that generally such things are accomplished transparently with flex layouts that automatically adjust to screen size
I doubt much work is being done on it, but presumably they at least make sure it works; I mentioned it because a few posts up (edit: you) mention testing (rather than initial design) as the reason why having multiple designs is so difficult.
> The mobile vs desktop versions you posted are likely the same codebase with minimal (if any) differences....
It's entirely possible that they're derived from a similar codebase at some point, but what reaches the browser is significantly different - it's barely responsive, based on user-agent, and appears to be significantly different obfuscated blobs of HTML, CSS, and JS.
For example, the feature set required to support the HTML page could be frozen and the APIs backing them stable with no need to change. So testing isn’t really necessary. Alternatively, there’s just API changes being made to remove dependencies on deprecated code and so the testing coverage comes from the testing that happens of that API surface through other means. Finally, it could be that the HTML page is even fully staffed to support emerging markets. That’s a different budget potentially than the budget for the “rich” UI.
Again, my point isn’t to argue over the specific business pressures and practices Google has for their email UI. This requires a level of knowledge I don’t think either of us possess. All I’m trying to do is illustrate that there could be all kinds of pressures why the system is the way it is, but dismissing it as “laziness” or “stupidness” on the part of the designer is itself a lazy and stupid conclusion to make without concrete evidence. I generally assume that’s not the case and look for the incentives/pressures those people are under until there’s overwhelming evidence those people are actually stupid/incompetent (and even then, the question becomes what structures, incentives, pressures were in place to put those people in positions they shouldn’t occupy).
It used to be that you could check the width and height of the viewport and say something like “320px wide? Must be a touch interface, deploy the big buttons”. Then tablets got big and it was like “1024px? Could be a laptop, but it’s probably an iPad, which has a touch interface, deploy the big buttons”. Then laptops got touch screens, then the Surface Studio came in and was like “HAHAHAHA”.
Now the game is “1920x1080? Could be a big tablet with touch, or a 1080p monitor without touch, or a non-maximized window on a Surface Studio with touch, or maybe it’s a monitor without touch hooked up to a laptop with a touchscreen and our window could get moved between them at any time...”
Nowadays, there’s no single reliable way to tell if a page is going to have to support touch until it gets a touch event, by which point you’ve already rendered the UI and it’s too late.
[0] https://developer.mozilla.org/en-US/docs/Web/CSS/@media/poin...
Not easily so on mobile - the need to zoom multiple times and scroll in 2 directions to read simple information is a PITA.
Money
The triggering event may be an email bounce. I get a lot of github notifications sent to my email, and the failure of just one/a few may trigger the reverification.
When this happens, you can spin up a temporary server and have a mechanism in place to redirect email so you don't go down when your provider does.
Losing incoming email is pretty much the worst case scenario when it come to configuration errors. It about as bad as not having backups, in that both cases results in unrecoverable loss of data.
> When this happens, you can spin up a temporary server and have a mechanism in place to redirect email so you don't go down when your provider does.
Use a commercial provider, but fall back to your own server when it goes down without changing your email address.
IMAP seems to be working, though.
Could be inconsistent though.
Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.
s/his/their
s/s\/his\/her/s\/his\/their/
In other words:
s%s/s\\/his\\/her/s\\/his\\/their/%s#s/his/her#s/his/their#%
Though with respect to 'ages' apparently it's been around since at least the 14th century but certain purists tried to stamp it out at various times (just like the singular 'you' which no one currently has grammatical issues with I hope).
https://public.oed.com/blog/a-brief-history-of-singular-they...
The same reason it ever mattered how you refer to people, politeness and respect. If someone you consider "him" asks you to refer to them as "her" it's like someone asking you to call them by their full name "Rebecca" instead of "Becky" or "Jonathan" instead of "Jon". If you like and respect them, you do as they request because things which matter to them matter to you, and being polite to them is important to you. If you ignore what they ask, call them what you want, you communicate that you don't respect them and don't want to be polite, that you want to dominate and 'win' instead.
> "Pronouns can mean whatever you want them to mean"
Only one way. A specific person asking you to use a specific pronoun for themselves is wildly different from you unilaterally and universally saying that all women should feel included by the word "him" because "him" has no meaning anymore.
'Singular "their" etc., was an accepted part of the English language before the 18th-century grammarians started making arbitrary judgements as to what is "good English" and "bad English", based on a kind of pseudo-"logic" deduced from the Latin language, that has nothing whatever to do with English... And even after the old-line grammarians put it under their ban, this anathematized singular "their" construction never stopped being used by English-speakers, both orally and by serious literary writers.'
If you make an honest mistake and somebody attacks/humiliates/whatevers you for it, they're the asshole. If you can't manage to have some bare minimum consideration for somebody who is their own person just as much as you are, you are the asshole. I see both of the above happening pretty regularly these days.
This stuff is a real gimme, so really the most charitable interpretation of your comment is that you are being disingenuous for edginess' sake.
Now that everyone's replaceable, the popular culture desperately tries to shift focus into arguing about pronouns and terms.
Watch out, this is a road to nowhere. Forcing others to use the right pronoun won't build up your retirement fund, but will distract you from worrying about not having one. And the fact that you care about it more than about your opponent's T-shirt color could be an indication that you are being manipulated to not think about the long-term things.
Thank you, sir, for elevating our collective level of discourse.
This is where it crosses from insightful into conspiracy theory territory for me. People seem perfectly capable of groupthink-deluding themselves. Why cheapen your argument by postulating some master manipulator when it's not necessary for the deeper point you're making?
It will only lead to people focussing the discussion to challenge this particular aspect, or them disregarding all you've said, instead of engaging with the actual meat of the argument.
It should not be a problem that gmail is "down". Unless this would be happening for more than a few days, noone would lose e-mail. It's a problem that it's not returning a temporary error code, but permanent one.
I think a lot of time and effort is spent categorizing errors from external systems into transient or permanent, and it's always kind of a one-off thing because some of them depend on the specifics of the calling application. It definitely takes some iteration to get it perfect, and it's very possible to make mistakes.
You don't have milliseconds. You can take quite some time to handle the client. 10s of seconds for sure. For example default timeout for postfix smtp client when waiting for HELO is 5minutes.
https://status.cloud.google.com/incident/zall/20013#20013003
Well I guess the thing is left unanswered for now is why the quota management reduced the capacity for Google's IMS in the first place.
Maybe we will know someday :)
* We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. (e.g. what if you're in a situation where rolling back could make the problem worse? we might be Google, but we don't have magic wands)
* Debugging new failure modes is a coin flip: maybe your existing tools are sufficient to understand what's happening, but if they're not, getting that visibility can in itself be difficult. And just like everyone else, this can become a trial and error process: we find a plausible root cause, design and execute a mitigation based on that understanding, and then get more information that makes very clear that our hypothesis was incomplete (in the worst case, blatantly wrong).
As Douglas Adams says, "The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair."
Here comes the poison pills!
However, I'd speculate that in this instance, when you get that .0001% problem, less hands on deck makes work from home aspects less easier. Akin to remotely fixing somebodies PC over standing behind them.
With that premise I'd speculate in this instance that whilst not the root cause, may of been a small ripple that led to that root cause and/or lead to a slower resolution than what would normally get.
Those speculations aside, it will only highlight what that some tooling needs to adjust for remote workers as does design and set-ups more. Water cooler talk is not just for gossip and a counter would be more regular on-line group socialising at a work level so that not only the companies but the workers can fully adapt and embrace the work medium; But so the kinks and areas that need polishing can be polished and made better for all.
Lastly, I'd speculate that I'm totally wrong and yet what I said may well anecdote with some out there and resonate with others.
When you operate at Google's scale then everything that can go wrong, will go wrong. Google does an amazing job providing high-availability services to billions of users, but doing so is a constant learning process; they are constantly blazing new trails for which there are no established best practices, and so there will always be unforeseen issues.
Yes, apps are highly distributed. Yes, roll-outs are staggered and controlled.
But some things are necessarily global. Things like your Google account are global (what went down the other day). Of course you can (and Google does) design such a system such that it's distributed and tolerant of any given piece failing. But it's still one system. And so, if something goes wrong in a new and exciting way... It might just happen to hit the service globally.
When things go down, it's because something weird happened. You don't hear about all the times the regular process prevented downtime... because things don't go down.
Sometimes it's a script responsible of deployment that will propagate an issue to the whole system. Sometimes it's the routing that will go wrong (for example when AWS routed all production traffic to the test cluster instead of production cluster).
https://support.google.com/mail/thread/6187016
Maybe time to switch to a more reliable provider.
Did you try pulling them down using the API tester?: https://developers.google.com/gmail/api/reference/rest/v1/us...
Some of the internal formatting that Gmail uses has changed over the years, so more likely than not the API that parses the stored message for display in the Gmail UI is just throwing some kind of error.
Either way my point is that this is a pretty serious bug and they haven't even acknowledged it! Not a good look.
How much do you hate it as an engineer when sales people make tech promises to customers without asking you? For comms people, engineers leaking info publicly feels the same way.
> Sharing inside info on an ongoing incident is a great way to get fired
You're not disagreeing.
1) Harmless to share 2) Will never be shared by PR teams
I don't see anything wrong with asking people to share what they can.
Very surprising it’s not localised.
That's a considerable difference.
If you're going to shoebox me into a particular timezone - tell me what it is, and let me change it.
Edit: Apparently it's browser settings, which means you need scripting enabled for this page.
Good thing gmail wasn't completely stupid and didn't try to forward that one, getting in infinite loop...
The images on different websites I've visited don't load (e.g. twitter, bbc etc). When they do they load, they load verrry slowly
Use dnscrypt.
Plus, how would he send tweets if no internet? I think he likes being able to use Twitter more than he likes being President.
For whatever reason, gp seems to be under the impression that it's Trump who would use this to his own advantage, which is not consistent with the prevailing conspiracy theories.
The "internet" deserves these outages to make people - and CEOs, CIOs, etc - realize that in-house ~~engineering~~ sysadmins used to exist for good reasons.
under 99.9% is a 3 day credit, and they're currently at 99.4%