These pages are so useless. For my bank (which has its outages) I use this page: [0], which immediately starts to spike when there is something going on, while the official status page [1] lags by hours at best and never reports anything at worst. There is probably some money to be made by allestoringen.nl ;)
how does it compare to observed reality earlier today?
----------
Incident began at 2021-12-08 08:40 and ended at 2021-12-08 10:20 (times are in Coordinated Universal Time (UTC)).
Date Time Description
Dec 8, 2021 10:40 AM UTC The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
Dec 8, 2021 10:31 AM UTC We're aware of a problem with Google Calendar affecting a significant subset of users. We will provide an update by Dec 8, 2021, 11:00 AM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.
Users can experience 500 errors when using the public APIs, displaying the web page or are unable to sync.
Please open Google Meet to see the calendar events.
Hah, I just did the same thing - before clicking through to the status page, I had a bet with myself that everything would be green. Imagine my surprise when indeed everything was green.
If companies like Amazon and Google are going to outright lie on their status pages, why bother having them at all?
Because somebody (customer) is monitoring the status page and measuring how long it was not green and hold you accountable for your service being down. So it‘s „better“ to update your status page afterwards when your legal team is all on board.
Technically, German should have them, but barely anyone bothers since they're not on a normal keyboard layout. Some word processors replace them automatically though.
That very much is what https://downdetector.com/ and https://downforeveryoneorjustme.com/ are, crowd-sourced downtime reports. HN as a whole doesn't care if some niche industry site is up/down, so would never surface past the 'new' page. It's also helpful to see if a problem is localized to a particular region.
I don't think status pages are so easy in practice. Me and everyone I know have not had any problems with google calendar today, so who is right? Together we're probably <0.001% of all google calendar users so I don't think our sample is very representative.
Presumably there is some internal threshold for when to flick the switch on the service dashboard, but without knowing anything about the scale of the outage or what the threshold is we're kinda shooting in the dark.
That's why you see the orange color instead of red. To indicate that 1. The survive is not fully functional, or 2. The service is not functional for some regions. And I've seen some companies add comments to each status to explain what's going on.
I'm sure the statuses are being very liberal at their reporting thresholds towards the end user. /s
Perhaps it would help if the dashboards gave a tiny inkling of transparency, like what the thresholds are so you could gauge the relative significance of your personal service outage. If my page is out and vendor shows green, then maybe those relying on my service ontop of vendor won't believe me as much when I pass blame to them and/or maybe the vendor isn't even aware of the issue, so I should contact them because I'm an edge case.
If my service is out and the status page shows red with a nifty "were working on it" I can very clearly show anyone I report to or who is dependent on my service the parent service outage causing the issue, maybe even link them to it. In addition, I know that I don't need to contact the vendors support because they're aware of the issue and actively working on resolving it. I'm sure their support staff will appreciate it that millions of people can avoid contacting their support service reporting the same outage and having to repeat the same information through more costly reporting mechanisms than a page everyone can simply observe.
Every power company I've been with for at least the past 10 years basically do this. If I'm out at work and come home to no power, I first load my power company's outage report site (service status dashboard). I check my area and magically I know if the power company is aware and likely working on it, I can even see their updated time estimates when service will be restored. Sometimes they even tell me what caused the issue (oh boy, information). I never even have to make a call.
Meanwhile if I'm sitting here sipping coffee and hear a loud noise down the street, power is out in the middle of the day and I look across the street and see the neighbor's interior lights on, check the power company's outage map and don't see it: I give them a call. Tada, I know my isolated outage will be addressed and the service provider is aware of the issue. I also know if it looks limited, I'm probably going to be a lower priority if there are multiple outages and limited maintenance staff, so I can sort of expect it may be several hours. I can head out and not waste my time and effort sitting around.
Let's treat critical internet service infrastructure (that's what 'cloud' wanted to be, so let them have all that entails) like we treat other critical service infrastructure. It works quite well.
The underlying issue I see is that my utilities are well regulated and watched over so there often is a lot of transparency about what's going on. Heck, they even have public hearings if they want to increase their prices. Meanwhile, private company offering some service isn't very regulated and has every incentive not to let you know to manage the perception of their image and reliability vs providing empirical actual data. So will we see any sort of transparency? I doubt it, just as I highly doubt the size of represntative truthfulness of any data they report.
I've used html-mode gmail since way back when. it's really nice, fast and simple. Unfortunately at some point several months ago, google stopped respecting having set html mode as default. the only way to get it now is to append "/h" at the end of the gmail link manually (or click the "use html mode" button while it loads, but that usually leads to it registering the click just as it finished loading, trashing a random email because the trash button is in the same location of the screen)
it's genuinely so frustrating, especially because if you use the html-link and then have to log in, it sends you back to javascript gmail again. I think this might actually be the thing that'll get me to switch from gmail, which I've been too lazy to do, but meaning to do for ages now
the worst part I think is that you can't middle click to open new tabs. I don't know if it's just me, but I constantly find myself wanting to cross reference two things that require multiple tabs, and SPAs never let you open anything in a new tab (even though they could if they bothered to implement it).
well, it still is around, it's just a hassle to get to these days, rather than loading automatically for those who opted into it. I doubt they even broke it on purpose, it's just one of those things that no one at google cares about, so it gets broken by something and no one notices or cares, and it'll never ever get fixed, because it's not like you can notify anyone at google who would care or have power to do anything.
I found a similarly great UI pattern the other day when leading a conference call. I clicked to toggle the attendee list, only to find that when I clicked a second time to hide it again, the controls had moved across so my mouse was over the hang-up button.
> SPAs never let you open anything in a new tab (even though they could if they bothered to implement it).
There's really nothing special about SPAs that prevent opening new tabs with middle click. All they need to do is use a href element instead of, e.g., a span with a click handler. For some inexplicable reason, gmail has opted for the latter even though most click targets update the url and navigate the user somewhere.
Of course for GMail that would result in taking 30s to load the email. If it wanted to have reasonable performance for opening the message in a new tab they would need to do some magic to transfer just the state required.
It's like everyone using these bloated JS front ends has forgotten that. Google, Reddit, Wells Fargo, Gusto, Twitter. They are actively breaking the built in browser features.
It's nowhere near as simple as that. Many SPAs don't even have URLs for referring to the items they display, so they'd also need to invent a complete URL for everything they might display and then also handle people visiting that URL. And if you change URLs in an SPA, you then need to interact with the browser history API, which, historically, has had a large number of disastrously bad bugs.
Things don't exist by default and every feature takes effort.
For gmail it is that simple. They've already solved providing unique URLs for all the internal app paths I bothered checking.
For SPAs in general, it's true that there is some extra work in gluing URLs and app state together, but there's no lack of libraries that make that process easy and painless.
Have you recently cleared your cache? For me gmail transfers 1.5 MB before displaying inbox and loading 40MB resources from cache. I am using default gmail config and latest chrome. I have 10s of thousands messages in inbox.
I once read a Twitter thread about the idea of an explicitly flaky AWS region.
It honestly sounds like it would be a really interesting itch to scratch - VMs with explicitly bad network, bad disk, etc. You could specify the severity and randomness of failures, or what days you want things to crash. You could even go nuts and corrupt memory, or randomly trigger GPFs or fault random instructions.
The best part is that the corruption could be cooperative, such that the initial handshakes of TLS connections always succeed, and things like database connections always establish correctly, and PID 1 never faults. So you always end up crashing somewhere obscure, infuriatingly annoying, and very interesting.
I understand that both NFS and rsync-without-SSH don't carry any form of authenticity checksum (IIUC NFS encryption does provide this if you set up Kerberos). Would be fun to corrupt random packets so they get the same TCP+IP checksum, eh...? ;)
And you could even go absolutely ballistic and borrow techniques from fuzzers like AFL to observe and trace long-running executions, and compute ways to trip a program up to get maximally divergent results. I wonder if you could build an inference model for that...
> 10:40 AM UTC The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
> Incident began at 2021-12-08 08:40 and ended at 2021-12-08 10:20 (times are in Coordinated Universal Time (UTC)).
This is a typical anti-bot Google infra CAPTCHA. Happens if you suddenly fire off many requests, well known to anyone who ever attempted to scrape Google or even did a very quick burst of searches while trying to solve a problem.
Last week Google Play was crashing on my phone over and over again (making the device an expensive paperweight over time since banking apps have mandatory updates)... I reinstalled and that fixed it for a short while and then after it started perma-crashing again I received the mail about mandatory 2FA, did that and since then no problems.
We need to move away from these big companies NOW!
Get your own HTTP, DNS and SMTP server (in that order), your own domains, host at home with fiber and lead-acid backup!
Make your ISPs enable static IP and open DNS + SMTP ports.
Next time that ever happens have something like `adb logcat | tee log.txt` running in the background so you can actually figure out what's crashing.
My old Android phone has gotten into a very sorry state, I badly need to factory reset it to fix it (eh, and maybe root it while I'm at it) once I get some data off of it - but the other day my bank app happened to be crashing, and I was able to determine it was because Chrome's WebView hadn't updated correctly. So (the Play Store has completely wedged itself and the app has disappeared, seemingly a catastrophically failed update) I grabbed an updated WebView APK from Ze Internetz, and yay, fixed.
Although yeah, I can't really recommend that particular strategy as a particularly secure first-class solution...
But watching logcat can be very useful. And you can at least wave the log info above your head like an annoying "I did my due dilligence" flag :P
87 comments
[ 3.0 ms ] story [ 42.2 ms ] threadhttps://downdetector.com/status/google-calendar/
But I'm getting 500 anyway ;)
[0]: https://allestoringen.nl/storing/rabobank/
[1]: https://www.rabobank.nl/particulieren/storing/
Re: banks, there is also the iDEAL status page [3].
[1] https://tweakers.net/nieuws/142053/storingswebsite-allestori...
[2] https://downdetector.com/enterprise/
[3] https://beschikbaarheid.ideal.nl/
how does it compare to observed reality earlier today?
----------
Incident began at 2021-12-08 08:40 and ended at 2021-12-08 10:20 (times are in Coordinated Universal Time (UTC)).
Date Time Description Dec 8, 2021 10:40 AM UTC The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
Dec 8, 2021 10:31 AM UTC We're aware of a problem with Google Calendar affecting a significant subset of users. We will provide an update by Dec 8, 2021, 11:00 AM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.
Users can experience 500 errors when using the public APIs, displaying the web page or are unable to sync.
Please open Google Meet to see the calendar events.
[1] https://github.com/insanum/gcalcli
*"The site is up, I just can't get to it. Fix my network!"
If companies like Amazon and Google are going to outright lie on their status pages, why bother having them at all?
>So it‘s „better“ to update your status page afterwards when your legal team is all on board
Is the initial double quote at the bottom something that is language specific?
Also, the upper one is “, not ".
And it‘s probably specifiy to German.
Presumably there is some internal threshold for when to flick the switch on the service dashboard, but without knowing anything about the scale of the outage or what the threshold is we're kinda shooting in the dark.
Perhaps it would help if the dashboards gave a tiny inkling of transparency, like what the thresholds are so you could gauge the relative significance of your personal service outage. If my page is out and vendor shows green, then maybe those relying on my service ontop of vendor won't believe me as much when I pass blame to them and/or maybe the vendor isn't even aware of the issue, so I should contact them because I'm an edge case.
If my service is out and the status page shows red with a nifty "were working on it" I can very clearly show anyone I report to or who is dependent on my service the parent service outage causing the issue, maybe even link them to it. In addition, I know that I don't need to contact the vendors support because they're aware of the issue and actively working on resolving it. I'm sure their support staff will appreciate it that millions of people can avoid contacting their support service reporting the same outage and having to repeat the same information through more costly reporting mechanisms than a page everyone can simply observe.
Every power company I've been with for at least the past 10 years basically do this. If I'm out at work and come home to no power, I first load my power company's outage report site (service status dashboard). I check my area and magically I know if the power company is aware and likely working on it, I can even see their updated time estimates when service will be restored. Sometimes they even tell me what caused the issue (oh boy, information). I never even have to make a call.
Meanwhile if I'm sitting here sipping coffee and hear a loud noise down the street, power is out in the middle of the day and I look across the street and see the neighbor's interior lights on, check the power company's outage map and don't see it: I give them a call. Tada, I know my isolated outage will be addressed and the service provider is aware of the issue. I also know if it looks limited, I'm probably going to be a lower priority if there are multiple outages and limited maintenance staff, so I can sort of expect it may be several hours. I can head out and not waste my time and effort sitting around.
Let's treat critical internet service infrastructure (that's what 'cloud' wanted to be, so let them have all that entails) like we treat other critical service infrastructure. It works quite well.
The underlying issue I see is that my utilities are well regulated and watched over so there often is a lot of transparency about what's going on. Heck, they even have public hearings if they want to increase their prices. Meanwhile, private company offering some service isn't very regulated and has every incentive not to let you know to manage the perception of their image and reliability vs providing empirical actual data. So will we see any sort of transparency? I doubt it, just as I highly doubt the size of represntative truthfulness of any data they report.
You can still access your calendar via https://calendar.google.com/u/0/embed?mode=week
This brings back the old UI (Google Agenda times).
I'm amazed by how little that page downloads. I'm getting less than 1MB here. Gmail downloads 20MB to display my inbox.
it's genuinely so frustrating, especially because if you use the html-link and then have to log in, it sends you back to javascript gmail again. I think this might actually be the thing that'll get me to switch from gmail, which I've been too lazy to do, but meaning to do for ages now
the worst part I think is that you can't middle click to open new tabs. I don't know if it's just me, but I constantly find myself wanting to cross reference two things that require multiple tabs, and SPAs never let you open anything in a new tab (even though they could if they bothered to implement it).
Joy.
There's really nothing special about SPAs that prevent opening new tabs with middle click. All they need to do is use a href element instead of, e.g., a span with a click handler. For some inexplicable reason, gmail has opted for the latter even though most click targets update the url and navigate the user somewhere.
It just seems lazy to me.
Things don't exist by default and every feature takes effort.
For SPAs in general, it's true that there is some extra work in gluing URLs and app state together, but there's no lack of libraries that make that process easy and painless.
Is any of that your email contents, or is it literally all JS and CSS?
AFAICT it's not as if it's downloading attachments ahead of time or anything.
Interestingly, it asked me to fill a captca test. Could it be an attack?
Certain customer may opt-out, but by default we should do global unplug once per year.
Outage should take a minimum of 8 hours
It honestly sounds like it would be a really interesting itch to scratch - VMs with explicitly bad network, bad disk, etc. You could specify the severity and randomness of failures, or what days you want things to crash. You could even go nuts and corrupt memory, or randomly trigger GPFs or fault random instructions.
The best part is that the corruption could be cooperative, such that the initial handshakes of TLS connections always succeed, and things like database connections always establish correctly, and PID 1 never faults. So you always end up crashing somewhere obscure, infuriatingly annoying, and very interesting.
I understand that both NFS and rsync-without-SSH don't carry any form of authenticity checksum (IIUC NFS encryption does provide this if you set up Kerberos). Would be fun to corrupt random packets so they get the same TCP+IP checksum, eh...? ;)
And you could even go absolutely ballistic and borrow techniques from fuzzers like AFL to observe and trace long-running executions, and compute ways to trip a program up to get maximally divergent results. I wonder if you could build an inference model for that...
Yeah, this sounds unreasonably fun.
> 10:40 AM UTC The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
> Incident began at 2021-12-08 08:40 and ended at 2021-12-08 10:20 (times are in Coordinated Universal Time (UTC)).
https://imgur.com/a/GQhvUen
Also, this isn't normal captchas where you can get sounds or stuff. These are completely inaccessible for many people.
We need to move away from these big companies NOW!
Get your own HTTP, DNS and SMTP server (in that order), your own domains, host at home with fiber and lead-acid backup!
Make your ISPs enable static IP and open DNS + SMTP ports.
My old Android phone has gotten into a very sorry state, I badly need to factory reset it to fix it (eh, and maybe root it while I'm at it) once I get some data off of it - but the other day my bank app happened to be crashing, and I was able to determine it was because Chrome's WebView hadn't updated correctly. So (the Play Store has completely wedged itself and the app has disappeared, seemingly a catastrophically failed update) I grabbed an updated WebView APK from Ze Internetz, and yay, fixed.
Although yeah, I can't really recommend that particular strategy as a particularly secure first-class solution...
But watching logcat can be very useful. And you can at least wave the log info above your head like an annoying "I did my due dilligence" flag :P
> expensive paperweight
Imagine if the expensive paperweight stops working as the mandatory 2FA.
I can imagine it now, workers eagerly awaiting the radio announcements over breakfast: "The following regions have Google datacenter outages..."