The style of the article is overly verbose to me, but the underlying incident is fascinating. Most of my workloads run on Windows Server, and having a monthly requirement to reboot due to updates is definitely annoying, but also a great way to force you to carefully orchestrate distributed system cold-starts from day one...
> This incident was costly, at $5M in lost revenue directly as a result of the outage, as revealed in a later earnings call. This was due to usage-based billing
No SLA whatsoever? Send Datadog your data, they lose it all, no discount or penalty but at least they don't charge you for storing it?
They didn't lose any data according to this article.
> For most customers, Datadog’s live functionality was restored within a day and a half, and all data was backfilled within two days, with no reported data loss.
We’ve recently signed up with datadog. I think their platform is good, but recently I’ve seen posts on a few platforms stating that they are known for up charging significantly and without warning. Wondering if anyone has experience with this and if we should have went with someone else
TLDR: Ubuntu unattended-upgrades and the systemd-networkd `ManageForeignRoutes=true` [1] default broke all Kubernetes networking globally, across multiple regions and cloud providers simultaneously, at 06:xx UTC. Recovery was difficult and slow due to the scale, particularly with cloud auto-scaling behavior kinda making things worse when all nodes failed simultaneously.
Please test how `systemctl restart systemd-networkd` affects your custom networking setup. It is quite likely to break.
I find it odd they hadn't seen this before, even with a moderate amount of route updates networkd will consume an excessive amount of CPU (even with foreign routes disabled it still processes netlink messages, annoyingly) - this is a pretty rookie thing to miss for a metrics company to be honest
> I’ll use Central European Summer Time (CEST) for time stamps, as the majority of Datadog’s engineering team is in this timezone. I’m also indicating US Eastern time (EST) for reference.
I can convert from UTC to my timezone (TZ) in my head, since logs are usually UTC. I know other engineers can do this, too. What I cannot do is convert from a random TZ to my TZ without looking it up. Clearly, the author knew this, which is why EST was added for an American audience.
Note the Datadog post-mortem is UTC [0], but the status page isn't [1]; even the EU one is ET [2].
So why are time stamps not being in UTC a thing?
Reasons I can think of:
* Status pages and post-mortems are not for engineers, but for management?
* Certain US-based companies use ET (i.e. EST or EDT) or PT (i.e. PST or PDT), and everybody else has cargo-culted it.
10 comments
[ 5.1 ms ] story [ 39.3 ms ] threadNo SLA whatsoever? Send Datadog your data, they lose it all, no discount or penalty but at least they don't charge you for storing it?
> For most customers, Datadog’s live functionality was restored within a day and a half, and all data was backfilled within two days, with no reported data loss.
Please test how `systemctl restart systemd-networkd` affects your custom networking setup. It is quite likely to break.
DataDog also published their own post-mortem: [2]
[1] https://www.freedesktop.org/software/systemd/man/networkd.co...
[2] https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...
I can convert from UTC to my timezone (TZ) in my head, since logs are usually UTC. I know other engineers can do this, too. What I cannot do is convert from a random TZ to my TZ without looking it up. Clearly, the author knew this, which is why EST was added for an American audience.
Note the Datadog post-mortem is UTC [0], but the status page isn't [1]; even the EU one is ET [2].
So why are time stamps not being in UTC a thing?
Reasons I can think of:
* Status pages and post-mortems are not for engineers, but for management?
* Certain US-based companies use ET (i.e. EST or EDT) or PT (i.e. PST or PDT), and everybody else has cargo-culted it.
[0] https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...
[1] https://status.datadoghq.com/history
[2] https://status.datadoghq.eu/history