What alternatives to Auth0 are worth looking into? Between this P0 (with no ability to check the status or file a ticket) and the Okta acquisition, I hesitate to continue using Auth0 as the default when spinning up new web apps.
Every year or so I check back in to see if Cognito has gotten any better. I hasn't. I'd love to use it, but some very basic things are just not done correctly. Yan Cui has a nice writeup on Cognito [0] that touches on some of its shortcomings.
My blood pressure is still coming back down from AWS having an all-day outage during a holiday week last year, because Kinesis had an issue and it turns out their entire infrastructure depends on that.
What's the point of such services? Every web framework worth its salt supports all common authentication schemes and offers an easy interface to write your own.
Keycloak is pretty battle-tested and rock solid although it is a bit of a behemoth operationally.
Outside of that they Ory ecosystem is really nice. We user Hydra, which is not a drop-in Oauth2 server but requires you to write several of the components yourself.
"Keycloak is pretty battle-tested and rock solid although it is a bit of a behemoth operationally."
Just looked. Java, Wildfly, Infinispan, Hibernate + A DB, network multicast if you want a cluster, your own separate load balancer, etc. So, a fairly large time investment if you aren't familiar with all of that.
Yeah, it's not the easiest thing in the world to get up and running but not quite as hard as it might seem at first look. If you are already using k8s then you can use https://github.com/codecentric/helm-charts/tree/master/chart... to deploy Keycloak fairly easily. If you're not using k8s then it is probably more of an undertaking.
Wow. The whole point of paying someone like Auth0 is to _not_ have this happen. This is basically their whole point, is it not? Really looking forward to the post-mortem, but I won't ever forget just how down this thing is right now.
Yup, exactly. We're actively working on ripping them out of our software, this is ridiculous. I knew we should have taken the time to make a fail safe...sigh...
It just means that I can point to someone else when the site goes down and can't do anything to get it back up. Like Auth0 today, or Microsoft a couple of weeks ago.
I suppose the point is that you get one 5h outage every other year _and_ you get to blame somebody else, rather than more regular ones because it's not really your core competency
We all know that all 3rd parties will have outages at some point, nobody's pretending otherwise
This is crazy timing -- my co-host and I just released a Podcast episode yesterday sifting through the details about the Auth0 database-related outage from 2018 ( https://downtimeproject.com/ ).
I'll be curious to see how much overlap or not there is with that previous outage. They wrote up a nice post-mortem back then, so hopefully we'll get another one this time.
> We have identified that this issue is coming from our database. However, we are still tracking down the root cause of the issue. We understand that this is an issue impacting the entirety of the environment, and the full Auth0 team is engaged in resolving this.
They're fast approaching the 95% availability mark for the month! Enterprise customers are already guaranteed 50% return on fees, I suspect they'll be giving out MUCH more than that.
We managed to get a reply from a C level. All we could get out of them was "something to do with our DB, but we don't know the root yet. Our fail-over process didn't work. This will never happen again".
Also, it only took them 2 and a half hours to admit it was their entire system instead of "a small subset of users" lol.
I don't think so, I think that it was a combo of malicious intent and some indexes that never got run. I guess you might call it a bad migration since indexes didn't get run, but that seems more like a catalyst than a root.
https://cdn.auth0.com/blog/20181128-Incident-RCA.pdf
Oof. Is there something about what they do that prevents you from having a completely separate second site? Or is this a case where "bad data" is being happily propagated to the redundant site?
As core as the service is, I imagined a panic button that reverted the database for site #2 to some specified point in time.
their dev is up, but prod is down for several hours mid day, INSANE. My users are asking me to have a failover... hard to explain they ARE the redundancy plan! Anyboy else have ideas here on how to explain to users?
Don't forget to request your credit per their SLA[0]. You have 10 days to request your credit, which by my calculations should be 10% of this months' charge. Not a fair trade for leaving us dead in the water for 4 hours, but SLAs in general are worthless.
This type of incident is exactly why I dislike identity federation as a service. Yes it's difficult to get right, and you open yourself up to additional risk and technical complexity to do the federation yourself, but simultaneously how many businesses are currently completely down and just sitting on their hands waiting for Auth0's engineers to fix their systems?
But how will you ever succeed if you don't outsource all but your core competencies while your core competencies simultaneously converge on banking/investment/midde-manning?
While this protracted outage has prompted some plans to get rolling with a self-hosted Auth0 substitute, bear in mind that this can't address users wanting to sign in with Google, Microsoft, LinkedIn, etc., where you'll still depend on their authentication service to work correctly, not ban your client ID, and so on.
Their status page is pretty depressing to read since all it really says, repeatedly, is "we're really sorry" and "we're working as hard as we can." You can just feel the abuse the person writing that must be getting from customers. I feel bad for them since they have no power to fix this and didn't cause the issue.
No root cause or resolution yet and it's been 4 hours. Doesn't bode well for getting this resolved soon.
Ha, whenever some 3rd party SaaS tool is down and I am obsessively refreshing their status page I'm always enraged with the lack of updates. But then later I think about what if I were one of the engineers trying to bring things back up. There is nothing worse than being heads down trying to get a mission-critical system back online and being constantly distracted by demands for "status updates."
What is the ETA on getting back online? Well, about 5 minutes longer than it would have been now that you pulled me away from what I was doing....
Good organizations have scheduled updates for critical outages. No one pesters anyone, and everyone knows that a five minute update is expected periodically, e.g. at the top of every hour.
Great organizations instantly spin up a response team which includes a communications liaison, who is technical enough to participate in the reponse, but is not presently leading it. They assist and communicate, either to the PM or to customers directly.
...and other organizations burn through humans until they get a handle on how to be better.
That said, in these days of AWS and GCE, you can go a loong time without running into the kinds of failures that used to cause critical, extended outages.
Yep, there is definitely a way to do it well and having scheduled, regular updates (as opposed to ad-hoc requests) is key. But when "the big one" comes then it is hard to maintain the discipline in that process.
And what is really frustrating is you see an error in a tool you depend on, try something to fix it (such as rebuilding a database index), then you sit back and wait without having any idea how long that process is going to take. Or if it is even going to fix the issue. So when someone asks how much longer, you really don't know. And there is a lot of quick pounding on the keyboard, followed by a long wait period where it feels like you aren't doing anything.
77 comments
[ 1.9 ms ] story [ 55.0 ms ] thread(famous last words)
[0]: https://theburningmonk.com/2021/03/the-case-for-and-against-...
Outside of that they Ory ecosystem is really nice. We user Hydra, which is not a drop-in Oauth2 server but requires you to write several of the components yourself.
Just looked. Java, Wildfly, Infinispan, Hibernate + A DB, network multicast if you want a cluster, your own separate load balancer, etc. So, a fairly large time investment if you aren't familiar with all of that.
Being in the cloud doesn't mitigate potential issues.
We all know that all 3rd parties will have outages at some point, nobody's pretending otherwise
I'll be curious to see how much overlap or not there is with that previous outage. They wrote up a nice post-mortem back then, so hopefully we'll get another one this time.
They still don't know what's going on. This is concerning. Already down for two hours of prime time.
Also, it only took them 2 and a half hours to admit it was their entire system instead of "a small subset of users" lol.
Oof. Is there something about what they do that prevents you from having a completely separate second site? Or is this a case where "bad data" is being happily propagated to the redundant site?
As core as the service is, I imagined a panic button that reverted the database for site #2 to some specified point in time.
I guess Okta did buy something core to the Saas ecosystem
[0]: https://auth0.com/docs/support/services-level-descriptions
Everyone make sure you've got detailed tickets into their service queue. Get your creds!
No root cause or resolution yet and it's been 4 hours. Doesn't bode well for getting this resolved soon.
What is the ETA on getting back online? Well, about 5 minutes longer than it would have been now that you pulled me away from what I was doing....
Good organizations have scheduled updates for critical outages. No one pesters anyone, and everyone knows that a five minute update is expected periodically, e.g. at the top of every hour.
Great organizations instantly spin up a response team which includes a communications liaison, who is technical enough to participate in the reponse, but is not presently leading it. They assist and communicate, either to the PM or to customers directly.
...and other organizations burn through humans until they get a handle on how to be better.
That said, in these days of AWS and GCE, you can go a loong time without running into the kinds of failures that used to cause critical, extended outages.