It didn't break anything for me. However, I am failing to understand the point of this design "refreshment". There is nothing new out there, just minor UI changes with no purpose?
The design refresh is actually one of the few I don't hate, I got it yesterday and it is decent. But the blank emails thing started today and this... is a problem.
"Things Happen", in production, to the best of us (and FM's pretty damned good at their job). Pretty sure someone's pager duty has been going off like mad.
A little over half an hour ago, the mail UI broke for me on Android, and then I panicked and went to desktop web and it broke there too. Also on different networks.
As far as I can tell, stuff from their CDN is 404-ing, and a JSON api POST request appears to be going in infinite loop with 200 OKs.
The webmail piece seems to be borked... Calendar, Files, Notes etc. are at least rendering.
There are a few different threads on this and now that things are in a stable place I'm going to cross post this to all of them!
The larger context is that we're making a major change to how we create IDs for email and mailboxes over the JMAP protocol. The old IDs are a UUID for mailboxId and the first 25 chars of the sha1 of the message for the emailId, prefixed by an 'M'. The new IDs are the createdmodseq for the mailbox prefixed by a 'P' (these are pretty short for most users) and a reverse counter of nanoseconds of the message internaldate (delivery time) for the emailId. This gives good storage density for offline and good data locality in databases for the email listings.
This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing.
This made MOST emails appear missing for most customers, clearly a very bad situation.
We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes.
Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1;
But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK.
...
P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.
11 comments
[ 5.4 ms ] story [ 38.6 ms ] threadAs soon as I realised that both my webmail and my phone app were buggered, it was probably not just a Me Problem.
And following FastMail's reply
> Hello Andrew! Can you please contact our support team so we can look into this for you? fastmail.com/support
They say:
> Don't have time. Consider my tweet the bug report.
Sorry but this asshole behavior. Bugs happen. No need to do public shaming and being rude to the company for that.
"Things Happen", in production, to the best of us (and FM's pretty damned good at their job). Pretty sure someone's pager duty has been going off like mad.
A little over half an hour ago, the mail UI broke for me on Android, and then I panicked and went to desktop web and it broke there too. Also on different networks.
As far as I can tell, stuff from their CDN is 404-ing, and a JSON api POST request appears to be going in infinite loop with 200 OKs.
The webmail piece seems to be borked... Calendar, Files, Notes etc. are at least rendering.
The larger context is that we're making a major change to how we create IDs for email and mailboxes over the JMAP protocol. The old IDs are a UUID for mailboxId and the first 25 chars of the sha1 of the message for the emailId, prefixed by an 'M'. The new IDs are the createdmodseq for the mailbox prefixed by a 'P' (these are pretty short for most users) and a reverse counter of nanoseconds of the message internaldate (delivery time) for the emailId. This gives good storage density for offline and good data locality in databases for the email listings.
You can see all the code for that in a handful of merge requests in the public cyrus-imapd repository on github at https://github.com/cyrusimap/cyrus-imapd/
Over the past few weeks, I've been helping out with the last bits of code modification, largely the changes on https://github.com/cyrusimap/cyrus-imapd/pull/5539 if you're interested.
This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing.
This made MOST emails appear missing for most customers, clearly a very bad situation.
We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes.
Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1;
Meanwhile, the real bug is fixed https://github.com/cyrusimap/cyrus-imapd/pull/5553 And a test has been written to prove it: https://github.com/cyrusimap/cyrus-imapd/pull/5554
But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK.
...
P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.