This kind of metastability _should_ be detectable through fairly simple load till failure testing and recovery monitoring. The 10x load cited is a red herring when pulling static config during recovery is their bottleneck
Hindsight is 20/20. This specific attack configuration highlighted a weak point.
What you don’t know about are all the other attacks that did not impact the service, because the way load testing was performed already lead to some mesures being taken.
Any recommendations of frameworks and learning materials? Ideally, something that could be integrated in the build process. I have mostly found some SaaS services (no-go) and Apache JMeter tutorials mostly aimed at QA (e.g., using GUI mode predominantly) so far.
I have used k6 in our CI/CD pipeline to do integration testing and then ran nightly actions to do smoke and load testing. It was easy to setup and it worked great.
The framework provides a clean way to define expectations and to produce data that can then be visualized in grafana.
>Due to the attack volume, a large number of processing nodes had to be brought online at the same time. Each of these nodes require a small amount of metadata that gets pulled from a central database
This is pretty interesting. Doesn't seem like the right scale for a small amount of metadata to topple over a DB. That's a pretty small load even if we're talking thousands of nodes.
A distributed system working against a single relational database, even if its just metadata, it's not only the ultimate anti-pattern but also sooo 2004...As this industry tends to shun the ’graybeards,’ there is never any institutional knowledge around. Every architecture seems to have to go through the same repeated adolescent struggles...
Your comment might be mixing up the Microservice pattern of a single database per bounded context.
From the moment one runs a system in two nodes, one being the application server and the other the database server, that system is distributed. Two nodes is all it takes.
>"Due to the attack volume, a large number of processing nodes had to be brought online at the same time. Each of these nodes require a small amount of metadata that gets pulled from a central database"
What is the nature of your argument? That because as long as we have two nodes, we call that a distributed system, the anti-pattern at scale is not valid?
> A distributed system working against a single relational database, even if its just metadata, it's not only the ultimate anti-pattern but also sooo 2004
Perhaps I wasn’t clear enough, but in your comment you make it sound that having multiple nodes interacting with a database is an anti-pattern, which is not, the discipline of distributed system is well researched and from the top of my head I risk to tell you that such anti-pattern is never mentioned, and the concept of a database rarely appears in DS, instead it focuses in a more abstract concept, which are the network nodes.
To support my argument, from the moment one decides to replicate nodes, most likely those same nodes will be connected to the same data source, because data replication isn’t as trivial as application replication.
In a Microservice architecture it can be considered an anti-pattern to have different bounded contexts using the same database.
Note I used the word relational databases. They must use connection pools because the database administrator, needs to protect the engine. Some DB engines even use a process per user connection, so it becomes very expensive on the server side. You need to multiplex thousands of connections, via a necessarily lower and upper bounded limited connection pool. The connection pool becomes the constraint. Use different, distributed, storage mechanisms, or basic Constraint Theory will come to bite you.
The attack volume is also mentioned to be only 10x normal traffic. For a DDOS attack, that doesn't seem like a lot... this type of spike could happen naturally to many web servers.
This Timeline of Events a bit scares me. It looks like, if someone hadn't decided to stop DDOS, deno would still had problems. I would expect to cut off all DDOS traffic on some gateway level, or so.
They addressed the issue by scaling up but the scale up itself ended up lagging behind the database. If that attack happens again presumably the bigger database + caching would mitigate it.
But that would incur a proportionally large cost on the infra I assume? Being able to cutoff DDOS at a gateway level by some sort of filter criteria would still be quite useful imo.
It’s hard for them to have two projects (the open source Deno and a closed source Deploy runtime that’s not actually Deno).
I’m wondering if this was open source rather than proprietary this sort of bug allowing a DDoS would have been caught before production. I know of multiple times in the past security issues were detected by the community and dealt in Node before they reached production.
It must be a challenge for the Deno team competing against the likes of Amazon or Google without the andvantage/burden of open source.
Not that I know, my reference was meetings with people from the Deno team (many of whom are friends) - I'm not sure it's documented but it can easily be verified. Just try to do `Deno.connectDatagram` on deploy for example and see it's missing.
I'm working at Deno, mostly on the open-source version so I can clarify a bit. The open-source code of Deno runs in Deploy, but some APIs are disabled and others are adapted for cloud use (ie: Kv works in CLI and Deploy, but is distributed on Deploy).
Under the hood Deno Deploy is just Deno with scale built in. It's obviously not a simple soundbite but I recently built the new Deno.serve API implementation and the same code effectively runs in the CLI and Deploy modulo what's necessary to get it working in the Deploy environment.
This seems like a really basic problem that would have been uncovered by some basic load testing. Y’know, the kind of thing you do (in addition to penetration testing, operational prep, etc) before you let customers use it. Fundamental stuff.
It was probably a known issue. It’s really just whether it’s worth addressing as a startup. Most early adopters are also willing to deal with a bit of pain.
Typically, it seems that the second major issue is when a company like deno needs together get serious about addressing known, but not profitable risks.
What are the best ways of stopping this without using cloud infrastructure these days? I’ve become too used to using the cloud offerings so I’m out of touch.
DDOS are not only still a thing, but getting worst over time. Look at the bottom historical map in this diagram - https://www.digitalattackmap.com/
You protect with a combination of services and some cloud providers will offer cost compensation for resources you might have to create to stay responsive during the attack.
42 comments
[ 2.9 ms ] story [ 86.1 ms ] threadThis kind of metastability _should_ be detectable through fairly simple load till failure testing and recovery monitoring. The 10x load cited is a red herring when pulling static config during recovery is their bottleneck
What you don’t know about are all the other attacks that did not impact the service, because the way load testing was performed already lead to some mesures being taken.
Was this a global outage for 90+% of deno deploy hosted projects? “A large chunk” doesn’t give me much faith that it was isolated to any one region
The framework provides a clean way to define expectations and to produce data that can then be visualized in grafana.
https://k6.io/
https://github.com/fluxninja/aperture
This is pretty interesting. Doesn't seem like the right scale for a small amount of metadata to topple over a DB. That's a pretty small load even if we're talking thousands of nodes.
Just a wild guess but the fact that connections are heavy in pg is a foot gun that tends to rear its head during large scale-up events
From the moment one runs a system in two nodes, one being the application server and the other the database server, that system is distributed. Two nodes is all it takes.
Perhaps I wasn’t clear enough, but in your comment you make it sound that having multiple nodes interacting with a database is an anti-pattern, which is not, the discipline of distributed system is well researched and from the top of my head I risk to tell you that such anti-pattern is never mentioned, and the concept of a database rarely appears in DS, instead it focuses in a more abstract concept, which are the network nodes.
To support my argument, from the moment one decides to replicate nodes, most likely those same nodes will be connected to the same data source, because data replication isn’t as trivial as application replication.
In a Microservice architecture it can be considered an anti-pattern to have different bounded contexts using the same database.
I’m wondering if this was open source rather than proprietary this sort of bug allowing a DDoS would have been caught before production. I know of multiple times in the past security issues were detected by the community and dealt in Node before they reached production.
It must be a challenge for the Deno team competing against the likes of Amazon or Google without the andvantage/burden of open source.
Under the hood Deno Deploy is just Deno with scale built in. It's obviously not a simple soundbite but I recently built the new Deno.serve API implementation and the same code effectively runs in the CLI and Deploy modulo what's necessary to get it working in the Deploy environment.
Keep throwing rocks if you or your org haven’t overlooked something in the system you work on.
Typically, it seems that the second major issue is when a company like deno needs together get serious about addressing known, but not profitable risks.
> The start-up payload will be cached across the globe
What are the best ways of stopping this without using cloud infrastructure these days? I’ve become too used to using the cloud offerings so I’m out of touch.
You protect with a combination of services and some cloud providers will offer cost compensation for resources you might have to create to stay responsive during the attack.
[0]: https://github.com/fluxninja/aperture
Load shedding is exactly what would prevent this in the future, scaling up capacity and adding a cache is only going to help buy some time.
Wish I could have a dollar for each time SREs duct tape rather than solve problems with systems thinking.
I believe you are getting downvotes because it's difficult to give a direct solution like you did without having more context on the matter.
> Look at all these super knowledgeable “engineers” down voting this.
Maybe people thought "look at this super knowledgeable 'engineer' shooting a one-off solution just by looking at an RCA"