4 comments

[ 1.0 ms ] story [ 18.1 ms ] thread
> We are currently experiencing a widespread outage in our US-1 Data center, and all hands are on deck to resolve it - we are truly sorry for the inconvenience and are working towards a timely resolution. The infrastructure that allows the configuration and resolution of our services is currently severely degraded, causing a number of customer-facing services to be disrupted. This results in high error rates in our web application and API, delays in metrics processing and disrupts alerting.

Wishing those devs all the best and happy hunting, they have an amazing product that saves my ass multiple times a day.

Must be crazy dealing with that many log events coming in. Hopefully they get it back up soon.
This is public postmortem levels of bad. It will be interesting to see what the root cause was.
Problems with service discovery and thundering herd problems according to a slack. Not sure what triggered the issue though.