I'm kind of curious what the source of the outage is.
It looks like most of their services are down, including their incident status page¹. The number of things I can think of that so many different services & are not a massively catastrophic failure is pretty small. Though that may be a lack of imagination on my part.
Doesn't have to be catastrophic failure. Most people just assume that such services are architected properly to be HA, but they usually aren't. They also aren't operated as HA usually, so things like a config change are rolled out globally to everything at once, and then everything breaks and they can't roll back, and you have total outage. Very common.
...and of course, they might not be down at all, it might just be (example) a route from the west coast got fucked in a BGP table so the west coast can't access it, and if that's where the majority of their users are that's all the comments you'll see.
We don't know for certain, but you may have missed Rogers [0] - which brought down cell phones (every function but wifi), debit transactions, and 911 services.
https://protonstatus.com/incidents/195#update-201
45 minutes ago : Our engineering team is working on a database performance issue which is impacting Proton services
Relational database woes. :( Probably re-indexing or slow queries or something. This is why companies with an RDBMS as a SPOF need a good DBA.
A minute ago StackOverflow was also throwing "Oops! Something Bad Happened!" to me on many different pages. I wonder if that's somehow related or just a coincidence.
Just remember, clicking on refresh and searching the net what is going on want make the problem go anyway faster. Few of us urgently depend on email just this hour. An even if you do setting your alarm to some suitable interval and just doing something else is much more productive. Just act like the Arab cliche taxi driver who takes the hands of the steering wheel when it gets tight and say Allah will steer (No insult to anyone, I have never visited any Arab country. Probably the story is not true anyway, but a nice anecdote to accept things you cannot change.)
When hackernews was down recently I wanted to report it as big news in our company chat. But it was rather late here and I had already closed all work stuff for the weekend. So I first slept and when I woke up everything was working again. Oh yeah, I missed to share the scoop with my colleagues, but I guess I'll manage without any damage to my professional reputation.
Edit: Reading post-mortem afterwards is a different story. If it's well written instead of some corporate communications emptyspeak one can learn something. No mistake happens only once.
>Few of us urgently depend on email just this hour.
I agree that you shouldn't be manically refreshing your email every 5 minutes, but my mail service being down for a few hours would definitely be a major inconvenience if I needed to get a 2fa code during that time. Same if I needed to look at my email to view my tickets for an upcoming event.
That sounds like an fragile setup if you depend on a single email account to get 2FA codes. I have them configured typically on my office workstation (reachable remotely if everything works), on my laptop, on my phone, and I have the static backup codes in my password manager which is also distributed over several indpendent machines.
Of course if you have to respond to incoming tickets within a guaranteed short timeframe that gets a problem if nothing can come in.
Make your risk assessment in advance: Can I live without it for a while, in the worst case even until I have an alternative provider? If so then just do so when an outage happens. If that's not acceptable for your case you need to have an independent backup before the first outage.
It's a misfeature using e-mail for 2FA login codes. TOTP is far more reliable. Any service doing e-mail only 2FA should be called out and questioned. Never assume a user has access to their email at all times.
Right, I must admit I had never heard of email being used for 2FA codes. Is that some kind of standardized protocol or something completely homegrown?
In the far past (and still today with 2 banks) I use hardware/smartcard based solutions. But everything vaguely IT world seems to use TOTP. They typically advertise it as Google Autenticator, but I use different FOSS implementations on both PC and phone and they have worked 100% smoothly.
> Right, I must admit I had never heard of email being used for 2FA codes. Is that some kind of standardized protocol or something completely homegrown?
If you don't have 2fa explicitly set up and browse with VPN enabled, you'll frequently get sites that want you to authenticate via email. Amazon does this, for instance.
It sure was an excellent purchase -- one of the most secure services available world-wide, operated in the EU with strong privacy and completely outside US jurisdiction.
You weren't able to collect your emails for a brief 5 minutes, but you'll live.
Hello from Switzerland :) We are not in the EU not just "technically" but as a fact. None of these agreement covers the exchange of private emails or metadata with the EU.
Irony is the people most likey affected by a protonmail outage would be political staffers and strategic PR firms who would suddenly need to use messengers subject to disclosure.
As someone taken out by the Rogers outage last week, I'd be concerned the infrastructure of western powers may be too fragile to project power in a seriously contested conflict.
1. why don't people selfhost email more? mailinabox is pretty painless to set up and would give you infinitely more security/privacy whatever you want.
2. why is email deliverability still an issue? why do you need to use third party email tools to ensure your email doesn;t go to spam? i am not concerned with newsletters/ads/ that sort of thing but a replacement to gmail/protonmail??
i have personally gone with miab like last year and beyond the first few weeks of gmail putting my emails into spam(then having to inform the other user to unspam it), things have been painless. i get to occasionally spin up the ssh to update the server but its fine. backblaze b2 backups ensure i am somewhat safe(er). can it be better? yes. is it ready to be used in production, aka a replacement to your regular paid/free email? you be the judge but i am sold on this idea
- when you send emails from MyWebsite.com, you want to make sure that Hotmail, Yahoo, Gmail, Proton Mail, etc users get your email.
- when you receive emails to your hi@mywebsite.com email, you don't want spam
V1 of email providers ostensibly solved this problem using lots of data as well as "supervised" machine learning (e.g. you telling the provider that a specific email is spam).
Self-hosting seems awesome if you're only handing out the email to a few people (probably a few hundred max?), you don't think it will be widely shared, and it's a slightly unique email that makes it hard for a script to guess (e.g. "hello@mywebsite.com" will be easily guessed but "john2@mywebsite.com" may not be)
- because google would prefer you use gmail so they can train their data models on you
- google would also prefer to have a monopoly on all email so that g-suite is necessary to reach your customers.
- because your IP address was at one point an e-mail spam server, but, that only really matters because of the above two reasons. Any nit, however irrelevant, will be used used to beat you into using gmail.
All anti-spam systems have non-zero false positive rate. It is hard to fix all FP in advance, but relatively easy to do this on case-by-case basis. When an FP affects some large sender it gets fixed. When it affects small senders - it is ignored because Big Corps don't care about small fish. They would be happy if all small mail servers will disappear and the market will be controlled by a few big companies.
It is technically possible to self-host email and send mail to O365/Gmail. They don't block 100% of small mail servers, only a fraction of them. But if you out of lack and their AI decided to block your IP, contacting their mail team is form very hard to impossible.
Mostly what I've found as the main issue here is that popular blacklists will blacklist entire IP blocks instead of individual IPs, so you have to not only have a clean IP but also be on at least a /24 with all clean IPs. This can be hard to manage when relying on outside hosting. And unfortunately, trying to get around it by using a cheap/free relay like Amazon SES or SendGrid doesn't work, as they fail to keep their IPs clean (if they're even trying).
Has to be related to this maintenance announced a couple days ago[1]
> "We're conducting a short database intervention on Sunday, July 10th, 2022, between 9:00 - 10:00 am GMT+2. During this time, users will be unable to access all Proton services for 2-4 minutes. We apologize for the inconvenience."
Rather ironically, the site they link to for the incident status updates ( https://status.protontech.ch/incidents/195 ) is not loading for me: "The connection has timed out"
I mean, no one cares if accessing mail is down, but it looks like their MX is down, the status page reports incoming mail has a ‘partial’ outage.
I don’t care if I can’t access my email for a while.
I very much care if email sent to me isn’t received. They should just be dumping all mail received at their MX’s to maildirs or something if it’s this bad and process them properly later. Losing data is not ok.
SMTP is literally built with this scenario in mind. As long as their servers aren't incorrectly reporting a success the sender will (or at least should) retry.
> They should just be dumping all mail received at their MX’s to maildirs
Not that it makes all that much difference, but this would break the encryption model for non-pgp inbound emails which are meant to get encrypted with the users public key on arrival.
I have no doubt that manual interception of these messages would be trivial with a court order anyway.
I just moved the domain for my small business over to ProtonMail (where I already hosted my personal email). The fact that I did that guaranteed that it would go down. :)
I was logged-in -- and composing a message -- a few minutes ago. First there was that cannot reach backup server message followed by a bunch (don't you just love technical jargon?) of messages saying "Could not..." and then lost the message when the site itself apparently went down. (Well, Firefox was seriously unhappy...) I was ready to blame my ISP again but no, this one is on proton mail. (It's pingable and traceroute is happy.)
I'm really glad that we didn't move our business over...
Yesterday we unfortunately experienced technical difficulties which meant that Proton services were offline intermittently throughout the late afternoon and evening European time. During this period, incoming email, outgoing email, and push notifications were also temporarily delayed. As of 2:45AM Geneva time, all emails and notifications are caught up, but we are continuing to monitor the situation in case of further faults.
We can confirm that no data and no emails were lost. Due to our redundant infrastructure, emails that were not immediately delivered were queued and automatically redelivered. We are still investigating the issue, but what we can share so far is that the technical difficulties were caused by a software update that was conducted over the weekend which adversely impacted the performance of certain systems. Unfortunately, this issue only appeared under under high loads which we were not able to fully simulate in testing. Furthermore, because this was a major software and operating system upgrade, there was no way to roll back the software update.
This result of this was an extended period of intermittent outages and performance issues while we tried to work around the issue. We have now put a number of safeguards in place, and will continue to improve the reliability of our infrastructure to guard against this type of issue in the future. We apologize deeply for this issue and thank you again for your understanding. Our engineering team is currently conducting a root cause analysis and for transparency, we will be sharing results on the Proton blog after our investigation is complete.
Seems like it’s not yet completely sorted… despite being impacted, i feel for the guys steering through this outage and bending backward! Good luck guys!
76 comments
[ 66.7 ms ] story [ 3060 ms ] threadIt looks like most of their services are down, including their incident status page¹. The number of things I can think of that so many different services & are not a massively catastrophic failure is pretty small. Though that may be a lack of imagination on my part.
1: https://status.protontech.ch/incidents/195
But they are down hard otherwise.
https://status.proton.me/
https://protonstatus.com/
...and of course, they might not be down at all, it might just be (example) a route from the west coast got fucked in a BGP table so the west coast can't access it, and if that's where the majority of their users are that's all the comments you'll see.
Like, for instance, a simple DNS or BGP misconfiguration that takes the entire network down?
Cloudflare (June 2022):
https://news.ycombinator.com/item?id=31823132
OVH (October 2021):
https://news.ycombinator.com/item?id=28849319
Facebook (October 2021):
https://news.ycombinator.com/item?id=28752131
Google (June 2019):
https://news.ycombinator.com/item?id=20077421
[0] https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...
https://nitter.net/ProtonSupport
also RSS feed:
https://nitter.net/ProtonSupport/rss
https://news.ycombinator.com/item?id=32023848 [Hacker News was down]
Reading the regular news like an idiot!
When hackernews was down recently I wanted to report it as big news in our company chat. But it was rather late here and I had already closed all work stuff for the weekend. So I first slept and when I woke up everything was working again. Oh yeah, I missed to share the scoop with my colleagues, but I guess I'll manage without any damage to my professional reputation.
Edit: Reading post-mortem afterwards is a different story. If it's well written instead of some corporate communications emptyspeak one can learn something. No mistake happens only once.
I agree that you shouldn't be manically refreshing your email every 5 minutes, but my mail service being down for a few hours would definitely be a major inconvenience if I needed to get a 2fa code during that time. Same if I needed to look at my email to view my tickets for an upcoming event.
Of course if you have to respond to incoming tickets within a guaranteed short timeframe that gets a problem if nothing can come in.
Make your risk assessment in advance: Can I live without it for a while, in the worst case even until I have an alternative provider? If so then just do so when an outage happens. If that's not acceptable for your case you need to have an independent backup before the first outage.
In the far past (and still today with 2 banks) I use hardware/smartcard based solutions. But everything vaguely IT world seems to use TOTP. They typically advertise it as Google Autenticator, but I use different FOSS implementations on both PC and phone and they have worked 100% smoothly.
If you don't have 2fa explicitly set up and browse with VPN enabled, you'll frequently get sites that want you to authenticate via email. Amazon does this, for instance.
You weren't able to collect your emails for a brief 5 minutes, but you'll live.
As someone taken out by the Rogers outage last week, I'd be concerned the infrastructure of western powers may be too fragile to project power in a seriously contested conflict.
1. why don't people selfhost email more? mailinabox is pretty painless to set up and would give you infinitely more security/privacy whatever you want. 2. why is email deliverability still an issue? why do you need to use third party email tools to ensure your email doesn;t go to spam? i am not concerned with newsletters/ads/ that sort of thing but a replacement to gmail/protonmail??
i have personally gone with miab like last year and beyond the first few weeks of gmail putting my emails into spam(then having to inform the other user to unspam it), things have been painless. i get to occasionally spin up the ssh to update the server but its fine. backblaze b2 backups ensure i am somewhat safe(er). can it be better? yes. is it ready to be used in production, aka a replacement to your regular paid/free email? you be the judge but i am sold on this idea
- when you send emails from MyWebsite.com, you want to make sure that Hotmail, Yahoo, Gmail, Proton Mail, etc users get your email.
- when you receive emails to your hi@mywebsite.com email, you don't want spam
V1 of email providers ostensibly solved this problem using lots of data as well as "supervised" machine learning (e.g. you telling the provider that a specific email is spam).
Self-hosting seems awesome if you're only handing out the email to a few people (probably a few hundred max?), you don't think it will be widely shared, and it's a slightly unique email that makes it hard for a script to guess (e.g. "hello@mywebsite.com" will be easily guessed but "john2@mywebsite.com" may not be)
- google would also prefer to have a monopoly on all email so that g-suite is necessary to reach your customers.
- because your IP address was at one point an e-mail spam server, but, that only really matters because of the above two reasons. Any nit, however irrelevant, will be used used to beat you into using gmail.
Mostly what I've found as the main issue here is that popular blacklists will blacklist entire IP blocks instead of individual IPs, so you have to not only have a clean IP but also be on at least a /24 with all clean IPs. This can be hard to manage when relying on outside hosting. And unfortunately, trying to get around it by using a cheap/free relay like Amazon SES or SendGrid doesn't work, as they fail to keep their IPs clean (if they're even trying).
> "We're conducting a short database intervention on Sunday, July 10th, 2022, between 9:00 - 10:00 am GMT+2. During this time, users will be unable to access all Proton services for 2-4 minutes. We apologize for the inconvenience."
[1]: https://twitter.com/ProtonSupport/status/1545698028895584257
I don’t care if I can’t access my email for a while.
I very much care if email sent to me isn’t received. They should just be dumping all mail received at their MX’s to maildirs or something if it’s this bad and process them properly later. Losing data is not ok.
Not that it makes all that much difference, but this would break the encryption model for non-pgp inbound emails which are meant to get encrypted with the users public key on arrival.
I have no doubt that manual interception of these messages would be trivial with a court order anyway.
I just moved the domain for my small business over to ProtonMail (where I already hosted my personal email). The fact that I did that guaranteed that it would go down. :)
I'm really glad that we didn't move our business over...
oof
>During this period, incoming email, outgoing email, and push notifications were also temporarily delayed
They say no data was lost but I'm not sure how they can know a sending mail server didn't give up trying to deliver a message to Proton
This is not to say there is no cause for concern, but SMTP was built for scenarios like this, at a time when these things happened far more often.
I wonder if there are any global standards, requirements recommendations about m the minimum retry count or timeout.
------
[1] https://protonstatus.com/
Their statement yesterday:
Yesterday we unfortunately experienced technical difficulties which meant that Proton services were offline intermittently throughout the late afternoon and evening European time. During this period, incoming email, outgoing email, and push notifications were also temporarily delayed. As of 2:45AM Geneva time, all emails and notifications are caught up, but we are continuing to monitor the situation in case of further faults.
We can confirm that no data and no emails were lost. Due to our redundant infrastructure, emails that were not immediately delivered were queued and automatically redelivered. We are still investigating the issue, but what we can share so far is that the technical difficulties were caused by a software update that was conducted over the weekend which adversely impacted the performance of certain systems. Unfortunately, this issue only appeared under under high loads which we were not able to fully simulate in testing. Furthermore, because this was a major software and operating system upgrade, there was no way to roll back the software update.
This result of this was an extended period of intermittent outages and performance issues while we tried to work around the issue. We have now put a number of safeguards in place, and will continue to improve the reliability of our infrastructure to guard against this type of issue in the future. We apologize deeply for this issue and thank you again for your understanding. Our engineering team is currently conducting a root cause analysis and for transparency, we will be sharing results on the Proton blog after our investigation is complete.