“… => Thus, we build a monolith service for each app, which is easy to deploy and maintain. No Docker, no Kubernetes, no dependencies, no runtime environment - just a binary file that can be deployed on any newly created VPS. …”
I don't use TablePlus myself. Are they talking about their marketing website? If so, then obviously thet wouldn't need to use Kubernetes. Are they talking about their application then I wonder how there can be no dependencies - don't it store data, log things, etc?
You can store data by connecting to a database, and you can store logs by either sending them to the system journal and having a daemon collect them, or sending them to whatever cloud you like using a logging library.
It's fine, really. Those database and logging services you can put in a docker if you like, but if you put them anywhere else it works just the same. A Postgres in k8s or a Postgres on a dedicated server is the same as far as the client is concerned.
I saw that deno did this but cool to see node picked it up too. I wish there was an option to run turbofan at build to generate the instructions rather than shipping the entire engine, but i guess that would require static deps and no eval, which can’t really be statically checked with certainty
Could you share how that can be done? I spent some time this year trying to pack a node tool into a single fat binary for a specific use case where we wanted a history of versioned executables - i.e a build job that needs to run specific versions of the packed tool in a specific order determined by external factors.
I tried Vercel pkg, Vercel ncc, nexe, and a few other tools I can’t remember right now. They all had issues with node v20, some dependencies, or seemed to not be maintained anymore. I ended up relying on esbuild as a compromise to get a fat script containing all sources and dependencies, tarballed with some static files we rely upon we can at least get versioned, reproducible runs (modulo the node env). Still not perfect, a single binary would be preferable
I don't remember the details, and cannot find my notes on vercel/pkg. But looking at https://github.com/vercel/pkg right now I see the project has been deprecated in favour of single-executable-applications
For Python, you could make a proper deployment binary using Nuitka (in standalone mode – avoid onefile mode for this). I'm not pretending it's as easy as building a Go executable: you may have to do some manual hacking for more unusual packages, and I don't think you can cross compile. I think a key element you're getting at is that Go executables have very few dependencies on OS packages, but with Python you only need the packages used for manylinux [2], which is not too onerous (although good luck finding that list if someone doesn't link it for you in a HN comment...).
How often the deployment model “copy a single binary to VPS via SSH and run it” is even used nowadays?
And with that still, you’d be much better served by using a more expressive and less painful to use language like C#. Especially if the type of use is personal.
A nice thing about modern cloud providers is their expensive bandwidth so a new vector of attack is simply downloading large files that they host. (except cloudflare)
Similar problem and similar-ish product 0]; we get DDoSsed a lot and I don’t know why. We had to put Cloudflare botfight to stop it. That works very well, but what do you do if CF doesn’t exist?
Depends who is paying for that? Self hosted that’s not a problem, but ‘serverless’ that’s usually costly at those levels, especially for worthless traffic.
assuming they're smeared equally across the whole month, that is. Eg if the majority of those requests kick off a job at midnight on the first of the month, it's a bit more to deal with.
Something like 300-400 API requests per second is not a heavy load for any reasonably designed API. Something like 300-400 static files requests per seconds is less than peanuts.
4 TB per month isn't really a DDoS attack, no? 4 TB per hour might qualify as a DDoS, but 4 TB per month is just 1.5 MB / second. 6 million requests per months are just 2 requests per second. I'd say the fact they run a monolith service isn't really relevant at this scale, especially as I assume Cloudflare handles most of the requests through caching them at the CDN level.
Sure, if every request triggers a very complex database transaction or computation, but if I understand correctly this is a simple file download endpoint that's probably cacheable.
I think OP means that the 6 million requests might not be evenly spread. They might only occur during 5 minutes of each day, for example. I don't know enough to know whether that's feasible.
Yes, you can rent a few dollar VPS from e.g. Hetzner (since Germany is mentioned in the blog post), and run a few wget commands in parallel in a loop on their 200MB setup file to easily reach 1TB a day.
For a company, this should definitely not be something to worry about. However, if I were able to single out individual IPs that are attacking me, then I would simply block them, report them (use the abuse form from the hoster of the attacking IP), and call it a day. This way, you can at least hope that the hoster will do something about it, either by kicking the hacker off its platform or, if it is some kind of service reflection attack, inform the victim to close the security loophole on their server and remove themselves from the botnet. If your attacks originate from a vast amount of different IPs from Russia and China, consider geoblocking.
On Hetzner, you receive an abuse email with the directive to respond appropriately if your root server or VPS is involved in some kind of abuse related issue. In larger companies this happens quite frequently. I'm not sure what would happen if you ignore such email.
Given that the vast majority of the web are Wordpress sites (or Drupal if you're a larger Org) that are on $5-20 per month multi-tenant hosts, have not installed caches, and speak directly to the database... the vast majority of sites on the internet can be knocked offline by merely doing 4 requests per second.
That sounds crazy, right? But yet that's where we are.
Context: I used to manage the DDoS protection at Cloudflare, and also the WAF, Firewall and some customer facing security projects, and we frequently either saw that web scraping took customers offline, or trivial and very low volume HTTP request rates took customers offline. In the early days we considered anything to be a DoS when it threatened our infra, but the threshold for customers is barely higher than a few concurrent and active users.
The big numbers always make headlines, but it's the small numbers which most people feel.
When I was setting up one of these $5 per month services(a simple php Web shop, with shopware if I remember correctly) I tested it with 10 concurrent users (5 req/s or so on average) and the $5.5 per month instance handled it just fine.
Yes, the instance had docker and was in an auto scaling group to be rebuilt if anything fails. There were 3 containers running with strict mem/cpu limits. Nginx reverse proxy (all in 128mb of ram), a mariadb sql server with a minimum of ~300mb of ram and up to 512mb if available and the php/Web host with 512mb ram reserved.
Mariadb was tuned and shopware was tweaked, but that's about it. Everything run fine on a 2 core, 1gb ram instance. (has, not "has been" because a year later the shop closed for other reasons). So the morale of this story is, sometimes a $5 vps or an instance is the correct answer.
The numbers given in my the article are 800k requests in five days for a static file, which is 1.85 requests per second and no database access. It's not even a DDoS by your own description here.
A Denial of Service attack could be mere KBs per a hour. If they trigger a heavy API endpoint on the target service, that may be enough to take the service down. The Distributed part just means multiple machines are attacking at the same time.
No part of a DDOS requires the throughput to be gigantic, although the big ones are typically the ones you will find in the news.
One possible aim of this attack is to either burn through the bandwidth quotum of the source servers, or to use so much bandwidth that it becomes unaffordable. This could be done very cheaply with just a single or few attacking machines. Most datacenters and hosting providers have bandwidth limits or start charging after a certain amount, and too often the company being attacked only finds out when they receive a bill they can't afford.
I have a website with a game download that has maybe 50 players total. Nobody uses it. I'm fairly certain that actual people downloading the game is maybe a few per month. Yet from this 2GB zip file I've had 5TB traffic over the last month and this has been going on for years.
This is no "DDOS", it's just misconfigured bots/crawlers going through all the links and being unable to cancel downloads.
I like this a lot: Why? Because the attacks are directed to someone who isn't bothered and wastes their own resources. I've been a Table Plus user for near a decade now and enjoy the simple but highly compatible software they provide.
A bit ingenious to say we do nothing when you have CloudFlare in front of your servers. Cloudflare by itself can automatically detect and handle DDoS without explicitly activating the Under Attack mode.
You can compile a jar to include all dependencies (like statically compiling C code), then you can just run `java -jar myprogram.jar` and it will work as long as the Java runtime is the same major version or newer than the version you compiled for.
That’s different from the runtime-free binaries produced by Rust and Go (binaries ship with tiny runtime) though. These are truly dependency-free, requiring only that you can execute ELF files.
Also those requests for the 200MB setup aren't even hitting your servers unless you have disabled caching for some reason, not that it'd be that hard to serve it directly.
When it's time for major shareholders and investors to 'exit' they don't want to market 'we did a simple setup', they want to be able to communicate twentyfive buzzwords incomprehensible to everyone directly involved.
It’s great that this isn’t hurting them but it leaves out a lot that makes me a bit nervous about this being taken as advice.
They’re advocating deploying a binary as preferable to using docker, fair enough, but what about the host running the binary? One of the reasons for using containers is to wrap your security hardening into your deployment so that anytime you do need to scale out you have confidence your security settings are identical across nodes.
On that, the monolith talked about here can be hosted on a single VPS, again that’s great (and cheap!), but if it crashes or the hardware fails for any reason that’s potentially substantial downtime.
The other worry I’d have is that tying everything into the monolith means losing any defence in depth in the application stack - if someone does breach your app through the frontend then they’ll be able to get right through to the backend data-store. This is one of the main reasons people put their data store behind an internal web service (so that you can security group it off in a private network away from the front-end to limit the attack surface to actions they would only have been able to perform through a web browser anyway).
>They’re advocating deploying a binary as preferable to using docker, fair enough, but what about the host running the binary? One of the reasons for using containers is to wrap your security hardening into your deployment so that anytime you do need to scale out you have confidence your security settings are identical across nodes.
There is no universe in which _increasing your attack surface_ increases your security.
If you’re using a typical docker host, say CoreOS, following a standard production setup, then running your app as a container on top of that (using an already hardened container that’s been audited), that whole stack has gone through a lot more review than your own custom-configured VPS. It also has several layers between the application and the host that would confine the application.
Docker would increase the attack surface, but a self-configured VPS would likely open a whole lot more windows and backdoors just by not being audited/reviewed.
You'd have to be utterly incompetent to make a self-configured VPS have more attack surface.
I have a FreeBSD server, three open ports: SSH with cert-login only, and http/https that go to nginx. No extra ports or pages for potentially vulnerable config tools.
Given the huge number of wide open production Mongo/ES/etc. instances dumped over the years, I wager having heard of ufw puts you among the top 50% of people deploying shit.
On the other hand. If by using containers it has become more feasible for your employees to use something like AppArmor, the end result may be more secure than the situation where the binary just runs on the system without any protection.
Considering the vast majority of exploits are at the application level (SQLi, XSS, etc), putting barriers between your various applications is a good thing to do. Sure, you could run 10 apps on 10+ VMs, but it's not cost efficient, and then you just have more servers to manage. If the choice is between run 10 "bare metal" apps on 1 VM or run 10 containers on 1 VM, I'll pick containers every time.
At that point, why are we making a distinction when we do run 1 app on one VM? Sure, containers have some overhead, but not enough for it to be a major concern for most apps, especially if you need more than 1 VM for the app anyway (horizontal scaling). The major attack vector added by containers is the possibility of container breakout, which is very real. But if you run that 1 app outside the container on that host, they don't have to break out of the container when they get RCE.
The VM/container distinction is less relevant to this discussion than you might think; both Amazon ECS and fly.io run customer workloads in VMs (“microVMs” in their lingo).
Containers don't really increase attack surface, it's all stuff provided by the OS anyway. Docker just ties it all together and makes things convenient.
Literally the entire guiding principle for security architecture for the past decade or even more has been that "there is no such thing as an indestructible wall".
I agree, perfection isn’t a realistic expectation. I also think effort spent building better defenses leads to fewer exploits over time than adding more of the same defenses. The marginal cost of bypassing a given defense is far lower than the initial cost to bypass a new defense
> One of the reasons for using containers is to wrap your security hardening into your deployment so that anytime you do need to scale out you have confidence your security settings are identical across nodes.
This is false. Or so you think your host is secured by installing Docker? And when you scale, how do you get additional hosts configured?
True is, when you use Docker you need to not only ensure that your containers are secure, but also your host (the services running your containers). And when you scale up, and you need to deploy additional hosts, they need to be just as secure.
And if you're using infrastructure as code and configuration as code, it does not matter if you are deploying a binary after configuring your system, or Docker.
Complexity is the criminal in any scenario. However, if we simply focus on a vanilla installation of docker, then the namespace isolation alone can be viewed as a step up from running directly on the os. Of course complexity means a vulnerability in the docker stack exposes you to additional risk, whereas a systemd svc running as a service account is likely to contain any 0day better.
> They’re advocating deploying a binary as preferable to using docker, fair enough, but what about the host running the binary? One of the reasons for using containers is to wrap your security hardening into your deployment so that anytime you do need to scale out you have confidence your security settings are identical across nodes.
There are tools that make "bare metal" configuration reproducible (to varying degrees), e.g. NixOS, Ansible, building Amazon AMI images.
I never understood how one “breaches an app through the frontend”. SQLi messes with your data store, natively (no RCE). XSS messes with other users, laterally. But how does one reach from the frontend all the way through, liberally? Are people running JavaScript interpreters with shell access inside of their Go API services and call eval on user input? It’s just so far fetched, on a technical level.
It just means that there is one big binary that does everything, instead of a bunch of microservices communicating over a fabric of some kind.
As someone who thinks microservices actually simplify a lot of things, especially in complex domains, the idea that a monolith is a choice makes me cringe a bit.
The idea of unnecessarily replacing nanosecond scale function calls with network communication that is five orders of magnitudes slower makes me shiver. Yeah, you can make a microservice that does one thing with a well-defined API and it's nice and clean. But you might as well make a module that does one thing with a well-defined API, and it will be so much faster because it's right there in memory with you.
If you’re replacing nanosecond function calls with microservices, you’re doing it wrong. It’s a specious argument.
In the domains in which I’ve worked, most services receive calls over the network, and go on to make database calls that also go over the network. So whether you do the routing inside or outside a monolith makes almost no difference to latency. And what’s more, with a front end like GraphQL, you can parallelise the work which reduces latency further.
Microservices have a lot of benefits relative to monoliths, but they aren’t a panacea any more than monoliths are. They’re a useful architecture for certain workloads and a poor fit for certain others.
But in my experience it’s quite a lot more difficult to maintain discipline over the long term with monolithic architectures, and that’s why I tend to prefer microservices attached to messaging architectures like NATS. YMMV, and that’s fine.
Speed isn't the main argument as I see it, rather complexity is.
How much of each of your microservices is boilerplate code? How much is outright copy-pasted? Microservices can be an invitation to write lots of lines of code, so that management sees that you're "efficient".
It's a lot about trusting your infrastructure, your development process, your overall plan with respect to state and communication etc. If you need to make it more complicated later, can you? Probably. If you see you could have done something simpler, can you de-complicate it? Sounds harder to me in general.
> If you see you could have done something simpler, can you de-complicate it?
this is why I think microservices are generally better - because they increase the friction to create coupling, and coupling creates complexity, and in particular makes it very difficult to de-complicate things. Put another way, microservices make coupling far more obvious, and the presence of strong coupling in a microservice architecture should be a massive red flag - one that's really easy to miss in a monolith.
The unfortunate thing about participating in these exchanges, however, is that it's impossible to explain ourselves in sufficient detail, so in the monolith vs microservice debate, we all end up just speaking across each other.
It's just that in my experience, the friction didn't do much to prevent coupling, because the friction was just that you had to write a lot of boilerplate code - not hard, just boring - and they were willing to do that rather than having a design meeting with me...
"To start explaining the microservice style it's useful to compare it to the monolithic style: a monolithic application built as a single unit. Enterprise Applications are often built in three main parts: a client-side user interface (consisting of HTML pages and javascript running in a browser on the user's machine) a database (consisting of many tables inserted into a common, and usually relational, database management system), and a server-side application. The server-side application will handle HTTP requests, execute domain logic, retrieve and update data from the database, and select and populate HTML views to be sent to the browser. This server-side application is a monolith - a single logical executable[2]. Any changes to the system involve building and deploying a new version of the server-side application."
It’s a kind of word which only makes sense as a negation to its antonym. Because if the antonym didn’t exist then it would just fade into the background as “normal”.
Public boasting as a mitigation strategy, that's got to be a new one.
Not entirely sure it's a wise approach given the deeply asymmetric infrastructure costs of DDoS attacks, especially if the attacker has access to a botnet.
[EDIT]:
in other words, there is a non-zero probability that the attacker, piqued by the boasting, might be able at the flick of a switch to increase the intensity of the attack by a factor 1M.
I'd hardly call that a DDoS attack: from the description given, the extra 8TB-or-so of monthly traffic seems to fall under "annoyingly pointless abuse of services"...
As long as such abuse doesn't cause monetary or resource exhaustion concerns, it's quite OK to ignore it, but stories like "whelp, turns out that 80% of the capacity of our auto-scaling fleet is not doing anything useful" are depressingly common enough to at least keep an eye on things.
My annoyance with this kind of abuse revolves mostly around logging: a majority of logs just showing the same set of hosts displaying the same pointless behavior over and over again. Again, not a huge issue if your log storage is cheap and plentiful (as it should be), but having some kind of way to automatically classify certain traffic as abusive and suppress routine handling of that is definitely a good idea.
It's also a lot harder than it sounds! I can't count the number of times I've added classification logic to my inbound SMTP server that should pick up on outright, silly abuse (of which there is a lot when dealing with email), only to have it triggered by some borderline-valid scenario as well.
Spending way too much time on going down successive rabbit holes is a great way not to get any real work done -- a great reason to outsource, or, if that's too much work as well or just too expensive, indeed just ignore the abuse, annoying though it is...
The TL;DR of my comment is "I personally enjoy implementing automated solutions to relatively-low-volume abuse, but as long as it doesn't cause you any capacity concerns, I fully understand ignoring it, since it's hard"
Using that as a reason to assign me responsibility for the state of the internet seems... slight hyperbole?
It wasnt directed at you as person, but as an idea. Not sure if you ever did abuse report, but they are mostly ignored. Thats the problem. Everyone just waves the hand like, it doesnt make capacity issues, we can ignore it. Sure, until its too late.
Maybe I am overly paranoid, but seems that old russian maxima is reasonable: Fight when they coming for cent, because when they will come to take dollar it will be too late.
So, funny story, a major reason why abuse reporting became pointless (unless done at the right level, i.e. when there is a direct and significant business relationship between the parties involved) is... abuse of the abuse reporting process!
Sometime around the dawn of this millennium, for example, 'consumer firewalls' at just about every OSI layer became a thing, and a lot of these had the great feature where they would automatically email WHOIS contacts for domains and IP blocks (plus all of their upstreams, for good measure) every time something bad happened, like receiving a single UDP packet on port 139.
Stuff like that, predictably, put a bit of a dent in the availability of useful technical contact information, and as much as I would like to go back to the "I have the beeper number of the guy who runs the national backbone" Internet, I'm afraid that Cloudflare is the best we can do these days, sorry.
Back to the topic at hand: "fight" on the 2024 Internet means refusing service to abusive parties as much as possible. That responsibility is best outsourced (see 'Cloudflare' above...), and a hard undertaking if you want to do it yourself without causing collateral damage (which, yes, Cloudflare also does, but at least you get someone to point at!).
Expecting to somehow get in touch (or worse, 'get even') with the myriad of bulletproof hosters (who simply don't care), admins of bug-ridden/misconfigured systems (who often don't even understand the issue) and assorted detritus is unproductive. And, as with any "the ideal amount of fraud is nonzero" discussion, that can be a hard pill to swallow, but a necessary one nonetheless.
Huh, interesting story. Can you point to some sources about that? What FW software vendors did it? I never heard about such dumb feature. Really. If I have rule that DROPs traffic, I do NOT care anymore whats really going on (unless its DoS).
> As long as such abuse doesn't cause monetary [...] concerns
8TB egress on AWS is $595 (taking into account 1TB free egress/mo), while 8TB egress on Hetzner starts at less than $10/mo. With DigitalOcean you'd pay $30 overage for 3TB on top of 5TB included in the s-4vcpu-8gb-intel, for example. 3TB overage is $15 with Linode. I think the article has a point.
The article (the point of which is: 'just ignore this', which is sort-of the opposite of the conclusion you seem to have gotten to) specifically mentions that their egress is free via Cloudflare.
But, sure, if you have public-facing services on AWS that have the ability to send large amounts of data on demand, absolutely make sure that you limit access to those! (E.g. using a unique download token that is only available from a separate rate-limited and valid-source-checking service).
What I was trying to say is that the article describes an architecture that takes cloud billing abuse attacks into account (they point out specifically that R2 is preferred to S3 due to egress cost structure) and this design is what partially allows them to ignore the light attack.
Most of the cloud architecture posts on HN either focus on how k8s/%your favourite new tool% is good for scale or detrimental to keeping complexity under control. And I think it's valuable for startups to consider cloud billing abuse attacks in addition to horizontal scaling concerns and complexity, which is what I referred to when I said the article has a point. As you wrote, rate limiting and extra checks could get the job done in a scalable deployment, so there is more than one way to keep cloud bill from an attack.
Yeah Goroutines are great. Then add something like WebRTC to your project that realistically tops out at 10000 listeners, and people wonder why Twitter Spaces is so buggy...
I guess this is an ad, so I'll bite: Why is the mac download button featured so centrally, while there appear also to be downloads for other platforms, too? It's not like that's a usual default.
Was hoping for something more swole dog worthy when reading the headline. Even though I agree with much of the advice, being behind Cloudflare is definetly not nothing.
Depending on the distribution of the traffic they might have survived well on VPS's without Cloudflare anyways, doesn't seem that large. Would be interesting to see more detailed stats of rps and how much (if any) Cloudflare stopped before they got it.
Russian layer7 ddos'es that I know of targeting Swedish companies have been large enough that major providers run into capacity problems and fall over (including Verizon, Azure Frontdoor, Cloudflare, GCP's Load balancer). This strategy would absolutely not work against those volumes.
Bragging about this has to rank up there as the worst idea in the world. If your hole argument is taunting would be attackers with your wallet - saying you're more overprovisioned than the traffic they can send, you're just threatening them with a good time. At another time in my life I'd take this post as an invitation, even, specially because the numbers shared are super low.
I've had 3 situations where my place of work was under DoS attack, in the 3 cases I managed to identify an email address and reached out asking why they are doing it, and if they want to talk about our backend. In 1 case, the "attack" was a broken script by someone learning how to program, the other two were real attacks and one of them just immediately stopped once they knew we knew who they were, the other actually wanted to chat and we emailed back and forward a bit.
99.99% of the time a DoS is someone who is bored. Talking to them tends to work.
Edit: there's some questions about the situations so I'll expand:
- The first was not a real attack, and they were doing the network calls through their authenticated API key. This was early days of a YC startup so of course there was no rate limiting in place. In this case I exchanged 2 or 3 emails and after they sent me their python script I sent them back a patch and they finished their scraping without bringing us down. Never heard from them again
- The second was at a different company, we were getting targeted to distribute email spam, because at the time we'd allow people to invite their colleagues as members of their account, and some people associated with casinos based out of Macau automated a way to spam their casinos by putting the URL in the name of the account, which went out in the email notification. I contacted one of the admin emails of one of the casinos I found and they stopped and disappeared. In this case we also locked all their accounts and prevented further logins + emailed them to reach out to support if they thought it was a mistake.
- The third one was more difficult, they weren't using any account, so all we had was network. At some point on the second day though they changed how they were sending some of the calls, and by mistake or not leaked their Telegram username. I installed telegram and talked to them, they trolled me a little bit, but stopped very quickly and didn't start it again. This one was very amusing to people in my company because I had told them this approach would work but a few of the big wigs didn't want me to do it (they didnt have any reason other than "obviously won't work to just talk"). I just did it anyway.
To be clear, you shouldn't reach out with some threats or how you're so good that you found them. My approach is of genuine curiosity, and my literal first message to the telegram person was:
"Hello, how is it going? I work at <companyname> and we're seeing a load of requests originating from your user here on telegram. Does this make any sense to you or do you think I might have the wrong person?"
>99.99% of the time a DoS is someone who is bored. Talking to them tends to work.
This is an overstatement. A great number are extortionists or state sponsored attackers. They're not interested in chit chat, except if it's to negotiate a price to stop the flood. Particularly not the ones commanding resources which are sufficient enough to make a dent in the operations of a substantial commercial entity.
How did you deal with these three customers, if that's not a secret?
RE: your tidbit on "don't threaten bad actors with good time", I disagree. If you can brag and demonstrate that their fleet of 10k machines can't touch a single service of yours and can't even make it pause then I'd say that sends pretty strong message to other potential bad actors.
Those bad actors have to be discouraged. I would get a very nice ego trip if I knew I can show the finger to a state actor, metaphorically speaking. But again, they should get the message that they are not as dangerous as they think they are.
Though I agree with other commenters that this traffic didn't seem as scary as the last 2-3 recorded attacks going through Cloudflare.
"If you can brag and demonstrate that their fleet of 10k machines can't touch a single service of yours and can't even make it pause then I'd say that sends pretty strong message to other potential bad actors"
You seem to fundamentally misunderstand the mindset of these attackers. This can just be about a challenge or a show of capabilities on the side of the attacker.
Telling them how big and strong you are will just trigger some of them to show you just how big and strong their botnet is. And there's always someone with a big enough botnet to bring you down.
> Telling them how big and strong you are will just trigger some of them to show you just how big and strong their botnet is.
And? Let them keep feeding more data to Cloudflare so they deflect them even easier next time around.
Maybe I do misunderstand their motivation. But you seem to think they can knock down anyone they want. To me that's observably false; even the record-setting DDoS attack through (or on?) Cloudflare that was something like 1.6Tb/s didn't seem to do much.
Let them flex. They provide us with valuable information for free. :)
This kind of proposal is one I hadn't ever really considered. I would be fascinated to know if others here have had similar experiences when seeking contact with malicious attackers.
Feeding the trolls might be the worst idea in the world but it's certainly the best way to harden your infrastructure through extreme real-world testing :)
In China 10 years ago, I've heard of stories that when someone got DDoSed and ransomed, they could pay to stop the attack (only relatively small amount of money). When they got DDoSed and ransomed again by another attacker, they simply told the previous attacker that they have already paid and the attack stopped.
It's going to a point where that directly find out the person behind it is cheaper than fix the bug. People nowadays don't pay respect to the hard working people anymore
What does a tool have to do when I'm purely searching for a certain MySQL/SQL syntax? TablePlus has done an awesome job of writing brief articles about different SQL syntax and its usage in TablePlus.
This is just a static marketing site for a desktop app. They don’t even have a discussion forum — feedback is handled by GitHub issues. Bragging about how simple their deployment is for a static marketing site and how it’s able to handle a static file being downloaded millions of times a day is super weird. And Cloudflare is doing all the mitigation work here (if that’s even needed for such a puny amount of traffic), not them.
If I were to be hit by such an "attack" myself I probably wouldn't even notice until Cloudflare sends me that monthly "X TB of data transferred, something close to 100% bandwidth saved" email.
I like the app as well but to me it also sounds like 'ChatGPT, create an unusal marketing post', it all doesn't make sense, even less for people who have experience with real ddos attacks.
204 comments
[ 2.4 ms ] story [ 246 ms ] thread“… => Thus, we build a monolith service for each app, which is easy to deploy and maintain. No Docker, no Kubernetes, no dependencies, no runtime environment - just a binary file that can be deployed on any newly created VPS. …”
It's fine, really. Those database and logging services you can put in a docker if you like, but if you put them anywhere else it works just the same. A Postgres in k8s or a Postgres on a dedicated server is the same as far as the client is concerned.
I'm mostly not following what is under a DDoS attack. Is it their web page mostly consisting of marketing material with static pages?
Yes.
Compare this to deploying python, node or php... Needless complexity.
If only running (and keeping running) a database server could be this straightforward!
I tried Vercel pkg, Vercel ncc, nexe, and a few other tools I can’t remember right now. They all had issues with node v20, some dependencies, or seemed to not be maintained anymore. I ended up relying on esbuild as a compromise to get a fat script containing all sources and dependencies, tarballed with some static files we rely upon we can at least get versioned, reproducible runs (modulo the node env). Still not perfect, a single binary would be preferable
Now you can use this native feature (not totally stable yet though) which I’ve been meaning to try https://nodejs.org/api/single-executable-applications.html
[1] https://nuitka.net/
[2] https://peps.python.org/pep-0599/#the-manylinux2014-policy
And with that still, you’d be much better served by using a more expressive and less painful to use language like C#. Especially if the type of use is personal.
0] https://flexlists.com
60 seconds per minute 60 minutes per hour 24 hours per day 30 days per month ~2.59 million seconds per month
One billion requests into 2.59 million seconds is 386 requests per second.
For a company, this should definitely not be something to worry about. However, if I were able to single out individual IPs that are attacking me, then I would simply block them, report them (use the abuse form from the hoster of the attacking IP), and call it a day. This way, you can at least hope that the hoster will do something about it, either by kicking the hacker off its platform or, if it is some kind of service reflection attack, inform the victim to close the security loophole on their server and remove themselves from the botnet. If your attacks originate from a vast amount of different IPs from Russia and China, consider geoblocking.
Cgnat is becoming common on home internet. You can share an IP with up to 128 other people.
That sounds crazy, right? But yet that's where we are.
Context: I used to manage the DDoS protection at Cloudflare, and also the WAF, Firewall and some customer facing security projects, and we frequently either saw that web scraping took customers offline, or trivial and very low volume HTTP request rates took customers offline. In the early days we considered anything to be a DoS when it threatened our infra, but the threshold for customers is barely higher than a few concurrent and active users.
The big numbers always make headlines, but it's the small numbers which most people feel.
Drupal, in particular, is notorious for having multiple layers of cacheing out of the box. Of course, you can always add some extra caches...
My favourite WP hosters:
- kinsta.com (for scaling & multi-sites)
- raidboxes.io (amazing customer support, usually within 15min, even on a Sunday)
Yes, the instance had docker and was in an auto scaling group to be rebuilt if anything fails. There were 3 containers running with strict mem/cpu limits. Nginx reverse proxy (all in 128mb of ram), a mariadb sql server with a minimum of ~300mb of ram and up to 512mb if available and the php/Web host with 512mb ram reserved. Mariadb was tuned and shopware was tweaked, but that's about it. Everything run fine on a 2 core, 1gb ram instance. (has, not "has been" because a year later the shop closed for other reasons). So the morale of this story is, sometimes a $5 vps or an instance is the correct answer.
No part of a DDOS requires the throughput to be gigantic, although the big ones are typically the ones you will find in the news.
One possible aim of this attack is to either burn through the bandwidth quotum of the source servers, or to use so much bandwidth that it becomes unaffordable. This could be done very cheaply with just a single or few attacking machines. Most datacenters and hosting providers have bandwidth limits or start charging after a certain amount, and too often the company being attacked only finds out when they receive a bill they can't afford.
This is no "DDOS", it's just misconfigured bots/crawlers going through all the links and being unable to cancel downloads.
Also Java jar files give you the same benefit.
You have to explain that one a bit more.
I think the word you're after is "disingenuous"
This sounds like a dream, both in the sense that it's wonderful, and that I'm not quite sure I believe it.
They’re advocating deploying a binary as preferable to using docker, fair enough, but what about the host running the binary? One of the reasons for using containers is to wrap your security hardening into your deployment so that anytime you do need to scale out you have confidence your security settings are identical across nodes.
On that, the monolith talked about here can be hosted on a single VPS, again that’s great (and cheap!), but if it crashes or the hardware fails for any reason that’s potentially substantial downtime.
The other worry I’d have is that tying everything into the monolith means losing any defence in depth in the application stack - if someone does breach your app through the frontend then they’ll be able to get right through to the backend data-store. This is one of the main reasons people put their data store behind an internal web service (so that you can security group it off in a private network away from the front-end to limit the attack surface to actions they would only have been able to perform through a web browser anyway).
There is no universe in which _increasing your attack surface_ increases your security.
If you’re using a typical docker host, say CoreOS, following a standard production setup, then running your app as a container on top of that (using an already hardened container that’s been audited), that whole stack has gone through a lot more review than your own custom-configured VPS. It also has several layers between the application and the host that would confine the application.
Docker would increase the attack surface, but a self-configured VPS would likely open a whole lot more windows and backdoors just by not being audited/reviewed.
I have a FreeBSD server, three open ports: SSH with cert-login only, and http/https that go to nginx. No extra ports or pages for potentially vulnerable config tools.
I guess no one knows how to harden an OS anymore so we just put everything in a container someone else made and hope for the best.
Are you suggesting that not opening the ports to any other services means they’re no longer a vulnerability concern?
That would be.. concerning.
At that point, why are we making a distinction when we do run 1 app on one VM? Sure, containers have some overhead, but not enough for it to be a major concern for most apps, especially if you need more than 1 VM for the app anyway (horizontal scaling). The major attack vector added by containers is the possibility of container breakout, which is very real. But if you run that 1 app outside the container on that host, they don't have to break out of the container when they get RCE.
The important thing is making walls indestructible, not making more walls. Interfaces decrease performance and increase complexity
(Some of) the reasons why you would do this are explained (I thought clearly) above. None of this is security through obscurity.
This is false. Or so you think your host is secured by installing Docker? And when you scale, how do you get additional hosts configured?
True is, when you use Docker you need to not only ensure that your containers are secure, but also your host (the services running your containers). And when you scale up, and you need to deploy additional hosts, they need to be just as secure.
And if you're using infrastructure as code and configuration as code, it does not matter if you are deploying a binary after configuring your system, or Docker.
There are tools that make "bare metal" configuration reproducible (to varying degrees), e.g. NixOS, Ansible, building Amazon AMI images.
What is it and how can one learn about it.
As someone who thinks microservices actually simplify a lot of things, especially in complex domains, the idea that a monolith is a choice makes me cringe a bit.
I mean they start out simple, but ...
Tell some others orgs I've worked at that microservices are simple and they would laugh.
But yes it depends on the complexity of your domain/org.
In the domains in which I’ve worked, most services receive calls over the network, and go on to make database calls that also go over the network. So whether you do the routing inside or outside a monolith makes almost no difference to latency. And what’s more, with a front end like GraphQL, you can parallelise the work which reduces latency further.
Microservices have a lot of benefits relative to monoliths, but they aren’t a panacea any more than monoliths are. They’re a useful architecture for certain workloads and a poor fit for certain others.
But in my experience it’s quite a lot more difficult to maintain discipline over the long term with monolithic architectures, and that’s why I tend to prefer microservices attached to messaging architectures like NATS. YMMV, and that’s fine.
How much of each of your microservices is boilerplate code? How much is outright copy-pasted? Microservices can be an invitation to write lots of lines of code, so that management sees that you're "efficient".
> If you see you could have done something simpler, can you de-complicate it?
this is why I think microservices are generally better - because they increase the friction to create coupling, and coupling creates complexity, and in particular makes it very difficult to de-complicate things. Put another way, microservices make coupling far more obvious, and the presence of strong coupling in a microservice architecture should be a massive red flag - one that's really easy to miss in a monolith.
The unfortunate thing about participating in these exchanges, however, is that it's impossible to explain ourselves in sufficient detail, so in the monolith vs microservice debate, we all end up just speaking across each other.
It's just that in my experience, the friction didn't do much to prevent coupling, because the friction was just that you had to write a lot of boilerplate code - not hard, just boring - and they were willing to do that rather than having a design meeting with me...
"To start explaining the microservice style it's useful to compare it to the monolithic style: a monolithic application built as a single unit. Enterprise Applications are often built in three main parts: a client-side user interface (consisting of HTML pages and javascript running in a browser on the user's machine) a database (consisting of many tables inserted into a common, and usually relational, database management system), and a server-side application. The server-side application will handle HTTP requests, execute domain logic, retrieve and update data from the database, and select and populate HTML views to be sent to the browser. This server-side application is a monolith - a single logical executable[2]. Any changes to the system involve building and deploying a new version of the server-side application."
Anyone have the cuneiform expression for 80/20?
anything but a flogged mare
Not entirely sure it's a wise approach given the deeply asymmetric infrastructure costs of DDoS attacks, especially if the attacker has access to a botnet.
[EDIT]:
in other words, there is a non-zero probability that the attacker, piqued by the boasting, might be able at the flick of a switch to increase the intensity of the attack by a factor 1M.
As long as such abuse doesn't cause monetary or resource exhaustion concerns, it's quite OK to ignore it, but stories like "whelp, turns out that 80% of the capacity of our auto-scaling fleet is not doing anything useful" are depressingly common enough to at least keep an eye on things.
My annoyance with this kind of abuse revolves mostly around logging: a majority of logs just showing the same set of hosts displaying the same pointless behavior over and over again. Again, not a huge issue if your log storage is cheap and plentiful (as it should be), but having some kind of way to automatically classify certain traffic as abusive and suppress routine handling of that is definitely a good idea.
It's also a lot harder than it sounds! I can't count the number of times I've added classification logic to my inbound SMTP server that should pick up on outright, silly abuse (of which there is a lot when dealing with email), only to have it triggered by some borderline-valid scenario as well.
Spending way too much time on going down successive rabbit holes is a great way not to get any real work done -- a great reason to outsource, or, if that's too much work as well or just too expensive, indeed just ignore the abuse, annoying though it is...
Using that as a reason to assign me responsibility for the state of the internet seems... slight hyperbole?
Maybe I am overly paranoid, but seems that old russian maxima is reasonable: Fight when they coming for cent, because when they will come to take dollar it will be too late.
Sometime around the dawn of this millennium, for example, 'consumer firewalls' at just about every OSI layer became a thing, and a lot of these had the great feature where they would automatically email WHOIS contacts for domains and IP blocks (plus all of their upstreams, for good measure) every time something bad happened, like receiving a single UDP packet on port 139.
Stuff like that, predictably, put a bit of a dent in the availability of useful technical contact information, and as much as I would like to go back to the "I have the beeper number of the guy who runs the national backbone" Internet, I'm afraid that Cloudflare is the best we can do these days, sorry.
Back to the topic at hand: "fight" on the 2024 Internet means refusing service to abusive parties as much as possible. That responsibility is best outsourced (see 'Cloudflare' above...), and a hard undertaking if you want to do it yourself without causing collateral damage (which, yes, Cloudflare also does, but at least you get someone to point at!).
Expecting to somehow get in touch (or worse, 'get even') with the myriad of bulletproof hosters (who simply don't care), admins of bug-ridden/misconfigured systems (who often don't even understand the issue) and assorted detritus is unproductive. And, as with any "the ideal amount of fraud is nonzero" discussion, that can be a hard pill to swallow, but a necessary one nonetheless.
8TB egress on AWS is $595 (taking into account 1TB free egress/mo), while 8TB egress on Hetzner starts at less than $10/mo. With DigitalOcean you'd pay $30 overage for 3TB on top of 5TB included in the s-4vcpu-8gb-intel, for example. 3TB overage is $15 with Linode. I think the article has a point.
But, sure, if you have public-facing services on AWS that have the ability to send large amounts of data on demand, absolutely make sure that you limit access to those! (E.g. using a unique download token that is only available from a separate rate-limited and valid-source-checking service).
Most of the cloud architecture posts on HN either focus on how k8s/%your favourite new tool% is good for scale or detrimental to keeping complexity under control. And I think it's valuable for startups to consider cloud billing abuse attacks in addition to horizontal scaling concerns and complexity, which is what I referred to when I said the article has a point. As you wrote, rate limiting and extra checks could get the job done in a scalable deployment, so there is more than one way to keep cloud bill from an attack.
Depending on the distribution of the traffic they might have survived well on VPS's without Cloudflare anyways, doesn't seem that large. Would be interesting to see more detailed stats of rps and how much (if any) Cloudflare stopped before they got it.
Russian layer7 ddos'es that I know of targeting Swedish companies have been large enough that major providers run into capacity problems and fall over (including Verizon, Azure Frontdoor, Cloudflare, GCP's Load balancer). This strategy would absolutely not work against those volumes.
> we keep things as minimal as possible
Wonder what's in that file that makes it need to be that large...
I've had 3 situations where my place of work was under DoS attack, in the 3 cases I managed to identify an email address and reached out asking why they are doing it, and if they want to talk about our backend. In 1 case, the "attack" was a broken script by someone learning how to program, the other two were real attacks and one of them just immediately stopped once they knew we knew who they were, the other actually wanted to chat and we emailed back and forward a bit.
99.99% of the time a DoS is someone who is bored. Talking to them tends to work.
Edit: there's some questions about the situations so I'll expand:
- The first was not a real attack, and they were doing the network calls through their authenticated API key. This was early days of a YC startup so of course there was no rate limiting in place. In this case I exchanged 2 or 3 emails and after they sent me their python script I sent them back a patch and they finished their scraping without bringing us down. Never heard from them again
- The second was at a different company, we were getting targeted to distribute email spam, because at the time we'd allow people to invite their colleagues as members of their account, and some people associated with casinos based out of Macau automated a way to spam their casinos by putting the URL in the name of the account, which went out in the email notification. I contacted one of the admin emails of one of the casinos I found and they stopped and disappeared. In this case we also locked all their accounts and prevented further logins + emailed them to reach out to support if they thought it was a mistake.
- The third one was more difficult, they weren't using any account, so all we had was network. At some point on the second day though they changed how they were sending some of the calls, and by mistake or not leaked their Telegram username. I installed telegram and talked to them, they trolled me a little bit, but stopped very quickly and didn't start it again. This one was very amusing to people in my company because I had told them this approach would work but a few of the big wigs didn't want me to do it (they didnt have any reason other than "obviously won't work to just talk"). I just did it anyway.
To be clear, you shouldn't reach out with some threats or how you're so good that you found them. My approach is of genuine curiosity, and my literal first message to the telegram person was:
"Hello, how is it going? I work at <companyname> and we're seeing a load of requests originating from your user here on telegram. Does this make any sense to you or do you think I might have the wrong person?"
That's it!
>99.99% of the time a DoS is someone who is bored. Talking to them tends to work.
This is an overstatement. A great number are extortionists or state sponsored attackers. They're not interested in chit chat, except if it's to negotiate a price to stop the flood. Particularly not the ones commanding resources which are sufficient enough to make a dent in the operations of a substantial commercial entity.
That sounds like a fascinating story. What did they want to chat about?
RE: your tidbit on "don't threaten bad actors with good time", I disagree. If you can brag and demonstrate that their fleet of 10k machines can't touch a single service of yours and can't even make it pause then I'd say that sends pretty strong message to other potential bad actors.
Those bad actors have to be discouraged. I would get a very nice ego trip if I knew I can show the finger to a state actor, metaphorically speaking. But again, they should get the message that they are not as dangerous as they think they are.
Though I agree with other commenters that this traffic didn't seem as scary as the last 2-3 recorded attacks going through Cloudflare.
It is also pretty good PR.
Telling them how big and strong you are will just trigger some of them to show you just how big and strong their botnet is. And there's always someone with a big enough botnet to bring you down.
And? Let them keep feeding more data to Cloudflare so they deflect them even easier next time around.
Maybe I do misunderstand their motivation. But you seem to think they can knock down anyone they want. To me that's observably false; even the record-setting DDoS attack through (or on?) Cloudflare that was something like 1.6Tb/s didn't seem to do much.
Let them flex. They provide us with valuable information for free. :)
Seems that they had some kind of alliance.
If I were to be hit by such an "attack" myself I probably wouldn't even notice until Cloudflare sends me that monthly "X TB of data transferred, something close to 100% bandwidth saved" email.
I like the app btw, can recommend.
To the contrary, I wish more people do this: The more people know that their overly complex infra is sub-optimal the better!