Also see the Netlify comment thread linked from one of the Reddit posts (https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-...), where one of their representatives specifically says that if you suffer a DDOS they will do nothing to stop you from getting a bajillion-dollar bandwidth bill, even if you're on the "free" tier.
That's my favorite thing about all these services. None of them have an equivalent of a stop-loss. They'll notify you IF you setup an alert, but that's it. Often times the alerts are way after the damage is done, with some billable metrics often lagging in reporting/alerting by hours. The default should be safe with easy options to increase the limits.
You can totally build a GCP function that will tear down your site if a billing alert hits the pubsub bus, but you can also set billing caps for a bunch of the APIs.
I don't really know although I can make up an answer but it contradicts
> None of them have an equivalent of a stop-loss.
Cynically, it's because they want to take your money. Practically, it's because a company that breaks your thing because you didn't pay, even if you asked it to, just isn't going to be a popular move.
It's probably quite difficult to boil it down to something as simple as an "off" switch that'll limit costs for them, meaningfully work for all services, not be abused, not be confusing, and not result in the customers that actually make you money accidentally getting important things shut off unexpectedly under predictable circumstances.
It's probably easier to make it so they can set up alerts and have a human on their end evaluate what steps are worth taking to cut costs while maintaining important availability.
From what I can tell Azure has more robust stop-loss options. But business considerations for the provider aside, there are absolutely tradeoffs between reliably limiting your spend and avoiding taking a business down because of a legitimate spike.
I remember trying AWS Glacier as a broke university student, thinking it was a cheap way to back up my data. I was wrong.
A decade later I played around with AWS, set up a free 1-core VPS. It was fine for a year or so (I ended up not using it at all), and suddenly I got a bill. It wasn't much, but it reminded me how these services make it very hard to be aware of your bills until you get them.
I was trying out Azure recently, and I found it too confusing to make any sort of quick risk assessment.
It's clearly just for people for whom "What's the worst that could happen?" is always a rethorical question.
Lambda and GCP Cloud Functions do have cost controls. You can specify the maximum number of instances that can be spun up at any given point.
I think the UX can make it more friendly to specify the types of traffic and cost you’re expecting but most erroneous spikes are the result of a customer misconfiguration or a legitimate spike in traffic.
It’s the former which are problematic for customers. They can request a credit and we (GCP) try and accommodate.
It’s not a feature designed for profit. But it also tends to fall below the cut line for prioritization to fix. But it’s in the backlog and doesn’t go unnoticed.
What you really want is prepaid instead of post-paid billing. Charge up the account with $100, and stop working and delete my data if that gets hit, with an auto renew option to recharge some amount every month.
That seems a little reductive. I don’t think it’s unreasonable to say, my current spend is ~13k a month. I should be able to set a 20k cap, such that non fixed billable like bandwidth or map tiles get paused while someone determines whether we’d like to proceed at that rate. There is a not so fine line between scaling for a rush/event and a resume generating event.
I wonder whether they are running out of money which then causes these things. Netlify used to be a very nice and unique service, but then came GitHub, GitLab and Cloudflare Pages who all provide roughly the same services for less money.
It used to be we will take your server down. Now its we'll DDOS your serverless website and leave you a 100k bill.
I'm not sure how sustainable such business model is. When you owned the server, you could unplug it. Now you have no way of knowing if somebody is going to hit your /api a million times per minute
I also prefer to get a (decaf) coffee, listen to some music while someone DDoS'd my VPS. I prefer to pay few $ / month for my VPS instead of paying thousands and "survive" the DDoS.
I pay $10/year for my VPS and host a WordPress Woocommerce store on it... It doesn't get much traffic, but it didn't take long to pay for itself either
I explicitly block everything in Russia (and China and Hong Kong) from accessing my servers, because otherwise the rate of SSH attacks is so high it actually prevents SSH from working reliably -- the connection pool fills up.
Just an FYI; I don't think it's that unusual. Blocking those geoips dramatically reduced my logging volume.
AWS guarantees protection from all DDoS attacks done at level 3-4 (Google aws shield). If someone calls your api million times then there is throttling.
So instead of DDoSing someone, you could make some not-so-large amount of requests to their APIs and instead of taking down their servers, you'll just take them down by bankrupting them with a huge invoice?
With default settings an attacker can run up-to 10000 req/sec on your api gw which would result in a sizable bill if left unnoticed. So with AWS you have to configure throttling and AWS protects you from low level ddos. How do you save yourself from a huge traffic bill in a VPS?
AWS also has WAF to protect from DDoS , it is expensive but may save a day if you urgently need a protection.
It really is. It decreases the risk and necessary KYC checks so a lot more people can get serviced that would be shown the door in a world where you can't sell a debt.
at least you wont get charged without your knowledge... which in the end, is the first step in getting this non-sense fixed, for a person... of course the government should fix it for everyone. This shit should probably be considered fraud anyways... companies get free passes all the time. My name (s9df898r32h) is my company, I should be able to steal whatever I want /s
Why are such large companies so incompetent though...
You will have to pay it AND a fee for fraud, and possibly their legal fees for tracking you down. You also will have a difficult time arguing that you were in good faith with this kind of shenanigans.
Bouncing checks are not a life hack to get stuff for free, they're a crime.
On the one hand I am glad to see sceptisim of serverless gaining mainstream acceptance.
On the otherhand, I fear it will just turn into an unreasoned backlash against technology that is often useful if not as often as its early proponents said it would be (See NoSQL).
Costing money is one thing, but not throttling a queue consuming serverless function and effectively DDOSsing yourself is the true horror story working with serverless :)
There's no end of stories of people discovering 10s to thousand in bill run up on AWS (ir Azure or GCP) because they foot gunned themselves and didn't set up alerts and/or automated service suspension using the cloud providers available tools.
Part of me thinks you shouldn't be allowed to provision EC2/S3/Lambda/whatever without first having set up a Cloudwatch alert and a rule to stop your service if your bill goes past expected/acceptable limits.
Of course Amazon (or Microsoft or Google) could make that _way_ easier than they do. It'd be nice if there was a simple form in your account setting where you could configure it to suspend services if your projected monthly bill passed $limit.
I think it's a more generic "horror story of working with other people's computers".
It’s an architecture failing. Setting maximums is critical. If you choose a platform that doesn’t support that more fool you. It’s implicitly easier to have a ‘circuit breaker’ in a traditional environment, but still warrants consideration. Load test for success and failure.
Because serverless 1) uses pay-as-you-go more often than traditional models 2) scales better to achieve massive usage.
If you DDoS my $5/month DigitalOcean VPS, it's not going to increase how much I have to pay DigitalOcean in terms of compute, and most likely the VPS will buckle under load before it can actually rack up massive transfer costs (since those are also pay-as-you-go for DO).
Something feels a bit off about the creator of this site also pushing a competitor. On one hand, they're upfront about it, and on the other.. they're upfront about it. At least it might prompt the companies in question to be a bit less scummy.
The link to the "competitor" is front and center on the site, so I imagine what you're really trying to say is that this is NOT a competitor for the following reasons: <missing>.
Andras (who created both Serverless Horrors and Coolify, which I imagine you're describing as the competitor here) is a pretty solid guy from everything I've seen and I've never seen him ask bait questions or engage in bad faith in discussions here or on other platforms.
Although he calls Coolify a Netlify 'alternative', it's 100% 'bring-your-own-server', and is not at all serverless. Coolify is basically a (very nice and featureful) fancy frontend to Docker with integration with Git.
Coolify is free if you want to install it yourself and don't mind the (minimal) tinkering needed to get it set up and keep it up-to-date, although you can pay a monthly fee to have your own servers managed for you, the main benefits are either to support continued development, or to access priority support. You still need to bring your own server, even if you're paying the management fee.
It's not a competitor to Netlify, it's an alternative.
Just to clarify, I didn't think that Coolify is a competitor.
What I was getting at is that we now have a thread with 4 replies: (1) you pitch competing product (2) what product? (3) Coolify (4) it's not competitor because ...
I just wanted to point out that comment #2 only seems to prompt question #3, whereas the helpful reply #4 created by yourself obviates the need for #2 and #3.
I agree, especially because I'm not even sure the pushed competitor solves the problem that's being highlighted here.
All the horror stories so far seem to be about people using a lot of metered resources, like bandwidth, and then getting billed for those resources.
With the pushed competitor you can self-host your Vercel or Netlify alternative on your own EC2 instance - but then if someone downloads 60 terabytes worth of data from you, you still get to pay for that bandwidth. You still get an unexpected large bill.
I don't think the Netlify case has happend due to serverless architecture. Yes, there are many technical issues with serverless architectures, but getting a huge bill for incoming traffic is independent from a serverless architecture. You could also put your own hardware in a colo and pay for traffic. If your data center provider does not protect you from DDoS and simply bills by the TB, then the same situation would occur. Of course, one could argue that the prices per TB are much cheaper in a colo situation compared to Netlify and therefore a 100k bill is very unlikely to occur, but then I would make this the main point: absurdly high traffic costs and no DDoS mitigation, and wouldn't put it behind a "serverless horrors" title.
If your stuff scales to infinity your money burn rate also scales to infinity. A single server in a colo can only do so much damage to your wallet before it’s saturated.
Sure, but according to the article, he received the 100k bill because of peak traffic of 60TB of bandwidth per day over several days. If you have a dedicated server in a colo, then a 10GbE connection is very common these days. With a 10GbE connection, you can serve 100TB of traffic in a single day. Hence, if the colo had the same prices as Netlify for traffic, then a single server with a 10GbE connection would be sufficient for such a high bill, no serverless and infinity scaling required, just absurdly high traffic costs.
Yup, this is exactly my point. It's not the serverless architecture, something you set up on Hetzner yourself with OpenFaaS (https://github.com/openfaas/faas), that is to blame here but the insane pricing scheme for traffic.
Correlation does not imply causation. There is no inherent reason why a serverless architecture would force a provider to have insane high traffic costs, hence it doesn't make sense to put blame on the term "serverless" for that.
On the other hand, if one could explain such causation, then I would be very interested in hearing about it.
Correct, but the effect on the wallet doesn't care if there is a causal relationship. And since my wallet is closer to me than a philosophical debate, the conclusion I reach in this case, is to move my site ;-)
I don't think infinite scaling is desirable. You probably want to say "if the value or rate of increase in xyz" metric reaches a certain level, don't scale further, or start scaling down. As a requirement for a serverless deployment, not an option.
You can set a hard budget on most cloud providers, including Azure etc. I actually did this thinking I was just setting a budget alert and that was a new and interesting way to interrupt production.
> You can set a hard budget on most cloud providers
Note that the submission references serverless providers, specifically netlify and vercel. I don’t think these support an easy to configure kill switch - in any case it wasn’t easy to configure in the scenarios listed on the page.
Everyone in this thread is talking about how this is not a Serverless problem, but a cloud problem. That's correct, but the point stands. Bandwidth is pure margin and to me, it's a bit scummy to a) charge this much for bandwidth and b) offer no tools to combat attacks that aren't in the customer's control. According to this thread[1], Netlify doesn't even have an option to take a site down for a while if it's under DDoS attack.
Netlify has multiple options to solve the problem: billing controls, request throttling, setting bandwidth limit, and waving of overages if it was a genuine DDoS, but it doesn't seem they are interested because that'd be like killing the golden goose. Just look at all the things bunny.net CDN offers [2].
It's disappointing how we acquiesce to explanations of cloud providers that are motivated by nothing other than greed.
Bandwidth charging models from hosts are crazy. They don’t model their upstream costs in any way. Transit providers charge based on 95th% and ports scale based on highest burst not volume passed. I’ve always hated it.
> The ddos attack was focused on a file on my site. Yes it’s partly my fault to put a 3.44MB size sound file on my site rather than using a third-party platform like SoundCloud. But still this doesn’t invalidate the point of having protection against such attacks, and limit the spending.
This is extremely sad. It's like we are taking steps backward. A 3.44MB should not be an issue and if it is, the answer should not be to host it elsewhere. If there is no other lesson learned here, it's that there is no such thing as 'free'. As others have said, there should be more done to prevent such large losses without some kind of limit.
I would also like to point out that VPS's are extremely cheap [1] and extremely easy to manage. They have automatic limits.
I remember a comment from this site. Something like: we must study how Amazon and other big cloud's marketing work so great to make this generation developers think that vps or selfhosting are hard.
IMO vps was easy before and even easier now to manage.
Please explain how you rotate ssh keys, store audit logs, backup (and test backup procedures), configure secure network between vpses such that your neighbor cannot eavesdrop.
For my personal stuff, when it comes to SSH key rotation and secure networks, I like to let Tailscale deal with both. It handles SSH authentication for you via your SSO provider, so there's no need to rotate keys.
(Aside: you shouldn't really be using SSH keys to begin with at anything but a small scale. SSH certificates are much more flexible)
You don't need to do any of that for hosting a content website.
The content & config are pushed by rsync/ssh from a git repo, so there's no need for backups. I can recreate a server in half an hour. I guess I lose the webserver logs, but I rarely look at them so I don't care.
A single server has plenty of bandwidth for a personal site, so there's only one EC2 instance and no secure network is needed. If I need more bandwidth, I'll use a load balancer but there's no need for secure connections between the load balancer & web servers because what's the eavesdropping threat model for a public content site?
Let's Encrypt seems to deal with https key rotation without manual intervention.
The cloud servers just have the usual ~cloud/.ssh/authorized_keys login setup, and I guess I rotate them every time a stronger crypto is recommended, which is 4ish times in 30 years.
If it is just a content website then maybe yes. Cloud complexity probably won’t worth it.
Still I can think of a corporate blog, and you have employees come and go then it became a problem even for a small website. Otherwise an angry admin can deface your website and damage your reputation.
All other things like secure net won’t apply for a small website, of course.
- you add your ssh public key to the hosting provider, so any new VM will have it automatically
- you use the snapshot service of your hosting provider for backups. If you have a database, run a cronjob that dumps it so it's in the snapshots as well. Alternatively use any backup tool to backup files to somewhere else
- you do not need a separate network for simple use cases. Just encrypt traffic if you have multiple servers, odds are you only have one here anyway.
* You don't have to rotate what doesn't get out. Limit ingress to relevant IPs reduces this surface area a lot.
* SCP to a system built for storage. Not really essential for many systems - system logs are fine.
* Every VPS provider comes with a backup check box.
* Tailscale is really simple.
This is how you protect one small server, you don't need cloud for this. But if you do use it for a small server you'll overpay a lot in relative terms and small amount in absolute terms.
> VPSs being “easy to manage” is a strong option full of assumptions.
There are definitely many footguns with managing a VPS but I think the threshold to get vaguely competent with a VPS is not really that far off with getting familiar with the average cloud platform - which comes with its own dangers, like the near-total inability to put an upward cap on fees that that person found out with Netlify recently.
Having a $5 VPS and knowing it's never going to cost your more than $5 might balance out a lot of things on the other side for a lot of people.
(And, as a bonus, it comes with the benefit of having a better idea of what is going on on the actual computer which is running your code.)
Platforms like https://coolify.io/ (which I have not tried, but looks interesting) seem to give you some of the abstractions that you get in cloud platforms to save you having to mess with too much low level stuff and become an expert in a billion separate systems.
If you have Debian with automatic updates that does most of the heavy lifting for you. The hardest problem I have is resisting the temptation to just install everything, because the cost to do it is capped at my VPS monthly fee.
So yep, it comes with a lot of assumptions. But so does everything!
a basic bitch static website takes three commands in the terminal and a very basic nginx config file to setup on a completely fresh ubuntu VPS. you could read a guide from 2012 and it would work perfectly fine.
is this harder than dealing with cloud 'platforms' and their ever-changing UIs, APIs, SDKs?
maybe? You go on Cloudflare's console, click on "pages", click on "new app", name it, drag your folder of website files in and you are done. automatic deploys are also available by linking a github account.
It's probably one less click to get a droplet turned up on Digital Ocean, and then you have to configure nginx or caddy (which is like five minutes if you've done it before, 30 if you haven't). Probably worth doing if it's your livelihood and you're afraid of Cloudflare disappearing, but if it's just a mess around... eh.
I won't run a VPS. I'm entirely capable of doing it.
Badly.
I'm a mediocre Linux admin, at best. The current environment is so dangerous, these days, that it's worth your life, to have your essential services run by a mediocre admin; even if I can convince myself that I'm the best (spoiler: I'm not).
I'll generally run shared hosting, or managed servers.
I can't speak for you, but it's really not too difficult to host some static content on a VPS. At the very worst, the VPS is compromised somehow, and you refresh it from the admin panel.
I need the whole enchilada: DB, Web Server, Dynamic Languages, etc. Also, for a shipping, production application, with hundreds of users; where privacy and security are of paramount importance.
It's easy, sure. It's easy to create an insecure server, that can be pwned. I know of which I speak. I have done just that.
"A man who holds a cat by the tail, learns a lesson he can learn in no other way."
Using private keys is not simple? sudo apt-get install ufw and ufw allow https is not simple?
The whole process takes maybe 30-60 minutes to setup for someone completely new and following guides.
Render's is simple, true, you just pay $300 for 1TB of bandwidth.
It's crazy how much Merchants of Complexity fooled devs into thinking that running your own server is complicated and you need to pay 1000x to save few minutes of your time.
> Any software engineer should be ... It really is simple.
I get that, a lot. It's the "No True Scotsman" of tech. This is also used as a way of validating college-style leetcode.
Let me introduce you to my GH Activity Graph[0]. See all that green? That's pretty much all coding in Swift; mostly in shipping apps and whatnot. There's a bit of PHP, for server-side stuff, but I like to spend a lot of time, coding frontend app stuff.
Every minute I spend, being a Linux admin, is a minute that I don't spend on executable code. I know that there's a number of folks, hereabout, that can code circles, around me, in Swift, and a few more, that can code circles around me, in PHP, but pretty much all of you, can run circles around me, in Linux admin. I'm not especially interested in competing, there; especially since a number of you are likely the folks that would Do Bad Things to my server, given an opening.
Every minute you spend on here, is a minute you don't spend on executable code as well, yet you're on here. Perhaps because spending every productive minute on coding is practically impossible.
You don't have to be a linux admin to be able to setup a properly secure server, nowadays it's quite trivial.
I bet you, that if you tried, you would be able to do so in less than an hour (at a relaxed pace).
We get lied to about the complexity of hosting by the cloud. I was able to host forums and game servers on VPSes when i was a young teenager, and I'm not technically gifted, it just wasn't complicated at all.
Well, we choose where to spend what free time we have. I like hanging here. There's really no pressure to perform. I'm too old to play insecurity-sop games.
I tend to like shipping stuff, which means taking Responsibility for its operation, maintenance and security.
That often means a lot of "not fun." My servers are working servers. They have data and capabilities that are important to a lot of "not-Chris" people. It's my job to make sure that they get what they are [not] paying for (I write free stuff). That can be a bit stressful, at times; especially when some bad actor is making life miserable for me. I'd much rather that be someone else's problem, where I can write an email that says "Make it so, Numbah One!", instead of spending two days, wrestling with config files, and CLIs.
Right up until someone runs a container with 8080:8080, which unfortunately bypasses ufw and the container is suddenly exposed to the entire internet .. :|
Oh man, that really sucks, to the point I would consider against using Docker for anything deployed. 5 years and such a basic security issue goes unfixed.
- packets alteration (iptable table mangle)
- applying filtering on behalf of a badly configured OS
So, if your case and if you want to prevent remote access to your database, you have a bad way: create a firewall rule to drop connections to tcp/3306
And you have a good way: configure your sql to bind to ::1
The firewall way requires two configuration (hence: complexity) and hide your intent : the mysql say : "I accept connections from everybody", and then the firewall say "I deny all connections".
While the good way is clear and sane : one component who say : "I only accept connections from localhost"
> A man who holds a cat by the tail, learns a lesson he can learn in no other way.
What a silly quote. “No other way” except all the other ones. Everyone watching the man and the cat will learn the same lesson. Everyone who hears the story will learn the same lesson.
I never held a cat by the tail, nor have i ever seen anyone foolish enough to attempt it, yet I am certain I know what happens next.
I think part of the point of the quote is that a man who would try to hold a cat by the tail is an example of someone who can ignore the experiences of other people.
People FAFO when they should know better all the time.
I think the person in the original article was, at least for the sound file.
> I need the whole enchilada: DB, Web Server, Dynamic Languages, etc. Also, for a shipping, production application, with hundreds of users; where privacy and security are of paramount importance.
It seems quite simple to me, you either learn security, or pay for the expertise of someone that has. You need to decide whether it's worth your time.
I would suggest one thing, though - even with the likes of <big cloud>, they will only provide security in limited cases, i.e. DDoS. Nobody at <big cloud> is going to make sure your application logic works correctly - they clearly won't even make sure you use their resources within sensible bounds.
>> The ddos attack was focused on a file on my site. Yes it’s partly my fault to put a 3.44MB size sound file on my site rather than using a third-party platform like SoundCloud.
And people wonder why there's only half a dozen websites any more and the "long tail" has vanished. You can post 3.44MB of audio to Facebook or Twitter and know that you'll never be billed for it.
Perhaps I'm overreacting, but I had my personal site on Netlify and decided to move it. I don't need anything more than a dumping ground for HTML, and Netlify was "good enough" (I wasn't aware that this issue existed)
What brought it closer to home was the characteristics of the recently affect site (same number of daily visits, not popular (a bit niche), etc) where similar to mine.
Moving is easy enough, mostly requiring a DNS update, since I prefer to build the HTML locally and then just dumping it somewhere.
What struck me as odd, when looking for an alternative is that almost none of the popular solutions Netlify, Vercel, Cloudfare (AFAIK) actually implement a spend limit.
Cloudflare has DDOS protection which you can set to basically paranoid and once a DDOS starts then everyone will get a captcha. This limits your spend quite effectively.
And also stop me accessing the site entirely. I don't know why, but every site with CF DDoS protection set to paranoid gives me infinite captchas that I can never pass.
I feel like cloud vendors overcharging for things they have the right to charge you are like a car mechanic that tells you you have to pay him a billion dollar for a routine maintenance check of your car because when he was fixing some small item, it broke and he had to get another one and put in in place, but then it broke for the same reason (that is not the mechanic's fault or the item constructor's), so he kept doing that on an infinite loop but after 2 weeks of him doing that without sending you any info, you went to the garage to check on him and told him to stop, so he stops and bills you for 999999999 items that are broken.
As a small user and not a large organisation, I'll never be able to read the full documentation on throttling every nook and cranny, I want to set a fixed spending limit that if reached, will kill the whole thing, even lose data if it must. But they won't do that because it hurts their bottom line if people are careful with budgets, and sometimes it's tricky to compute billing ahead of time.
Well there are differences. The mechanic from your example caused the situation. Netlify didn't and it might be hard to distinguish between unwanted traffic and a SaaS having overnight success.
A more accurate allegory would be if you told the mechanic to do whatever anyone who knows the registration plate asked, and he did so.
I think the biggest Cloud horror, probably not serverless, is that we have accepted to pay for inter-AZs traffic just in case the Cloud provider fails. In other words, they charge (and a lot) for a mitigation for the problems they might have.
The creator of the blog is Andres who is also building coolify[0], a heroku/netflify self-hosted alternative. I wanted to give a shoutout to him and his product. I've been running it for a few months and love it. Really was has been kinda the secret sauce of ease of use for me to start self-hosting things like changedetector, jdownloader, vaultwarden, etc.
It also has a pretty nicely growing community where people are contributing new templates (I added one for Syncthing) and helping each other debug.
The only issue I've had with it is things kinda fall over if you run out of disk space, which happened when I was running on an instance with just 10GB storage. So a little better alerting or prevention around that would be great but otherwise it has been pretty solid.
Presumably the costs for a platform are passed on to the platform's customers from that platform's cloud provider (with profit). When a spike happens due to, say, a DDOS, does the platform first have to negotiate with the cloud provider to waive the cost?
Presumably the costs for a cloud provider are passed on to the cloud provider's customers from that cloud provider's operating costs (with profit). When a spike happens due to, say, a DDOS, does the cloud provider first have to negotiate with its downstream suppliers (e.g., energy, network peering, etc.) to waive the cost?
...
How far down the chain does this go? Or maybe insurance steps in at some point -- or every point -- to break the chain?
Not necessarily - there are some costs we just eat. However in the event of outside actors having a reputable CDN in place to mitigate these things is worth your time and money.
Silly question, but while you're still at the making (expensive) mistakes stage, why not set up a cheap on-prem box that can be switched off/trashed with ease? When you're satisfied that you've fixed all the obvious weaknesses, move the software to The Cloud. Or am I missing the point?
These are just cost nightmares, what I love (hate) is when said serverless gives you the any kind of bizarre errors and some solutions require you to do some arcane "saw it on stackoverflow sub-comment".
Oh and it's not like you can jump into the console to debug.
I gave up after not even vercel could figure it out and merely told me that I should replace said package.
The thing I don't understand is why people go for cloud route for the simple sites, when there are options like Github static hosting or self hosting using secure tunnels from Cloudflare or Ngrok?
The risk presented by these horrors stories greatly outweigh possible complications from setting something quickly on your own.
Don't Netlify/Vercel run on top of the likes of AWS/GCP? That means they need to markup on those already inflated hyperscaler markups for data transfer.
I agree with the consensus here that the cost issues are a cloud problem, not serverless. Ironically, something like AWS lambda has a default 1,000 simultaneously activated lambda limit, so the platform defaults to throttling.
Ye, especially with something like Netlify or Vercel, I'd bet the vast majority of the costs are CDN bandwidth, and it never even touches the functions themselves. Could just as easily be in this situation with AWS CloudFront and an S3 bucket.
I have seen AWS Lambda generate large bills even with the 1000 concurrent executions limit, but with the help of upstream services. My memory of it is hazy, but I think over a weekend, a function with some dodgy retry logic managed to generate something like 21TB of CloudWatch logs.
173 comments
[ 3.1 ms ] story [ 200 ms ] thread> None of them have an equivalent of a stop-loss.
Cynically, it's because they want to take your money. Practically, it's because a company that breaks your thing because you didn't pay, even if you asked it to, just isn't going to be a popular move.
It's probably easier to make it so they can set up alerts and have a human on their end evaluate what steps are worth taking to cut costs while maintaining important availability.
A decade later I played around with AWS, set up a free 1-core VPS. It was fine for a year or so (I ended up not using it at all), and suddenly I got a bill. It wasn't much, but it reminded me how these services make it very hard to be aware of your bills until you get them.
I was trying out Azure recently, and I found it too confusing to make any sort of quick risk assessment.
It's clearly just for people for whom "What's the worst that could happen?" is always a rethorical question.
It's not cheap to restore though, at about $90 minimum per TB.
So it's best as part of a 3-2-1 system, not for frequent restores.
I think the UX can make it more friendly to specify the types of traffic and cost you’re expecting but most erroneous spikes are the result of a customer misconfiguration or a legitimate spike in traffic.
It’s the former which are problematic for customers. They can request a credit and we (GCP) try and accommodate.
It’s not a feature designed for profit. But it also tends to fall below the cut line for prioritization to fix. But it’s in the backlog and doesn’t go unnoticed.
(Former PM on GCP Cloud Functions)
gcloud run deploy ..... --max-instances=N
I'm not sure how sustainable such business model is. When you owned the server, you could unplug it. Now you have no way of knowing if somebody is going to hit your /api a million times per minute
Just an FYI; I don't think it's that unusual. Blocking those geoips dramatically reduced my logging volume.
"would be a shame if somebody read a bad review about your biz, pay us to remove it"
to Netlify
"would be a shame if somebody DDOS'd your serverless website, subscribe to our DDOS protection plan"
AWS also has WAF to protect from DDoS , it is expensive but may save a day if you urgently need a protection.
Nowadays everyone push you to add a card to open free* account. Such disposable cards is a solution to this issue.
I just ignore any horseshit that goes to collections outright
Why are such large companies so incompetent though...
Bouncing checks are not a life hack to get stuff for free, they're a crime.
On the otherhand, I fear it will just turn into an unreasoned backlash against technology that is often useful if not as often as its early proponents said it would be (See NoSQL).
There's no end of stories of people discovering 10s to thousand in bill run up on AWS (ir Azure or GCP) because they foot gunned themselves and didn't set up alerts and/or automated service suspension using the cloud providers available tools.
Part of me thinks you shouldn't be allowed to provision EC2/S3/Lambda/whatever without first having set up a Cloudwatch alert and a rule to stop your service if your bill goes past expected/acceptable limits.
Of course Amazon (or Microsoft or Google) could make that _way_ easier than they do. It'd be nice if there was a simple form in your account setting where you could configure it to suspend services if your projected monthly bill passed $limit.
I think it's a more generic "horror story of working with other people's computers".
It's a cloud problem.
Edit: Well, if you have small cloud servers they might go down under an attack. And if you pay only outgoing like in AWS that's good for you.
It's either a billing or an architecture problem.
Either let me throttle requests to prevent overbilling, or cut me off when I hit a certain value.
If you DDoS my $5/month DigitalOcean VPS, it's not going to increase how much I have to pay DigitalOcean in terms of compute, and most likely the VPS will buckle under load before it can actually rack up massive transfer costs (since those are also pay-as-you-go for DO).
> It's a cloud problem.
It's a serverless problem because it can be mitigated if using the cloud, but not mitigated if it is serverless.
I have a cloud instance. I don't have this problem and never will. If I had serverless instead, I will have this problem.
(I created serverlesshorrors)
Otherwise this seems like a bait question:)
Although he calls Coolify a Netlify 'alternative', it's 100% 'bring-your-own-server', and is not at all serverless. Coolify is basically a (very nice and featureful) fancy frontend to Docker with integration with Git.
Coolify is free if you want to install it yourself and don't mind the (minimal) tinkering needed to get it set up and keep it up-to-date, although you can pay a monthly fee to have your own servers managed for you, the main benefits are either to support continued development, or to access priority support. You still need to bring your own server, even if you're paying the management fee.
It's not a competitor to Netlify, it's an alternative.
What I was getting at is that we now have a thread with 4 replies: (1) you pitch competing product (2) what product? (3) Coolify (4) it's not competitor because ...
I just wanted to point out that comment #2 only seems to prompt question #3, whereas the helpful reply #4 created by yourself obviates the need for #2 and #3.
All the horror stories so far seem to be about people using a lot of metered resources, like bandwidth, and then getting billed for those resources.
With the pushed competitor you can self-host your Vercel or Netlify alternative on your own EC2 instance - but then if someone downloads 60 terabytes worth of data from you, you still get to pay for that bandwidth. You still get an unexpected large bill.
60 TB
On Hetzner (VPS provider): $1. On Netlify: $33,000
On the other hand, if one could explain such causation, then I would be very interested in hearing about it.
Correct, but the effect on the wallet doesn't care if there is a causal relationship. And since my wallet is closer to me than a philosophical debate, the conclusion I reach in this case, is to move my site ;-)
I’m guessing many hobby users would set that to $100, end of story.
I don’t understand why it’s so unpopular to offer that. I’d imagine the providers would benefit too in the long run.
Why does this not exist after years (more than a decade since EC2) of cloud computing?
Because it is not good for the VC investors.
Note that the submission references serverless providers, specifically netlify and vercel. I don’t think these support an easy to configure kill switch - in any case it wasn’t easy to configure in the scenarios listed on the page.
[0] https://www.web3isgoinggreat.com/
There used to be a whole host of specific fuckup like list site, sadly I can't find them anymore.
Netlify has multiple options to solve the problem: billing controls, request throttling, setting bandwidth limit, and waving of overages if it was a genuine DDoS, but it doesn't seem they are interested because that'd be like killing the golden goose. Just look at all the things bunny.net CDN offers [2].
It's disappointing how we acquiesce to explanations of cloud providers that are motivated by nothing other than greed.
[1]: https://answers.netlify.com/t/limiting-bandwidth-traffic-to-...
[2]: https://support.bunny.net/hc/en-us/articles/360014190440-Und...
Netlify just sent me a $104k bill for a simple static site | https://news.ycombinator.com/item?id=39520776
This is extremely sad. It's like we are taking steps backward. A 3.44MB should not be an issue and if it is, the answer should not be to host it elsewhere. If there is no other lesson learned here, it's that there is no such thing as 'free'. As others have said, there should be more done to prevent such large losses without some kind of limit.
I would also like to point out that VPS's are extremely cheap [1] and extremely easy to manage. They have automatic limits.
[1] https://lowendbox.com/blog/1-vps-1-usd-vps-per-month/
IMO vps was easy before and even easier now to manage.
(Aside: you shouldn't really be using SSH keys to begin with at anything but a small scale. SSH certificates are much more flexible)
2. Often there would be a cPanel plugin/extension/app/config value (if the hoster enabled it for you) that would just do for you what you needed.
The content & config are pushed by rsync/ssh from a git repo, so there's no need for backups. I can recreate a server in half an hour. I guess I lose the webserver logs, but I rarely look at them so I don't care.
A single server has plenty of bandwidth for a personal site, so there's only one EC2 instance and no secure network is needed. If I need more bandwidth, I'll use a load balancer but there's no need for secure connections between the load balancer & web servers because what's the eavesdropping threat model for a public content site?
Let's Encrypt seems to deal with https key rotation without manual intervention.
The cloud servers just have the usual ~cloud/.ssh/authorized_keys login setup, and I guess I rotate them every time a stronger crypto is recommended, which is 4ish times in 30 years.
Still I can think of a corporate blog, and you have employees come and go then it became a problem even for a small website. Otherwise an angry admin can deface your website and damage your reputation.
All other things like secure net won’t apply for a small website, of course.
- you add your ssh public key to the hosting provider, so any new VM will have it automatically
- you use the snapshot service of your hosting provider for backups. If you have a database, run a cronjob that dumps it so it's in the snapshots as well. Alternatively use any backup tool to backup files to somewhere else
- you do not need a separate network for simple use cases. Just encrypt traffic if you have multiple servers, odds are you only have one here anyway.
There are definitely many footguns with managing a VPS but I think the threshold to get vaguely competent with a VPS is not really that far off with getting familiar with the average cloud platform - which comes with its own dangers, like the near-total inability to put an upward cap on fees that that person found out with Netlify recently.
Having a $5 VPS and knowing it's never going to cost your more than $5 might balance out a lot of things on the other side for a lot of people.
(And, as a bonus, it comes with the benefit of having a better idea of what is going on on the actual computer which is running your code.)
Platforms like https://coolify.io/ (which I have not tried, but looks interesting) seem to give you some of the abstractions that you get in cloud platforms to save you having to mess with too much low level stuff and become an expert in a billion separate systems.
If you have Debian with automatic updates that does most of the heavy lifting for you. The hardest problem I have is resisting the temptation to just install everything, because the cost to do it is capped at my VPS monthly fee.
So yep, it comes with a lot of assumptions. But so does everything!
is this harder than dealing with cloud 'platforms' and their ever-changing UIs, APIs, SDKs?
It's probably one less click to get a droplet turned up on Digital Ocean, and then you have to configure nginx or caddy (which is like five minutes if you've done it before, 30 if you haven't). Probably worth doing if it's your livelihood and you're afraid of Cloudflare disappearing, but if it's just a mess around... eh.
add your config, use chatgpt if you want, save `nginx -s reload` and bam! you're good to go, practically forever.
Badly.
I'm a mediocre Linux admin, at best. The current environment is so dangerous, these days, that it's worth your life, to have your essential services run by a mediocre admin; even if I can convince myself that I'm the best (spoiler: I'm not).
I'll generally run shared hosting, or managed servers.
I need the whole enchilada: DB, Web Server, Dynamic Languages, etc. Also, for a shipping, production application, with hundreds of users; where privacy and security are of paramount importance.
It's easy, sure. It's easy to create an insecure server, that can be pwned. I know of which I speak. I have done just that.
"A man who holds a cat by the tail, learns a lesson he can learn in no other way."
- Mark Twain
Use private keys. Use a firewall (ufw is really simple) and only expose your reverse proxy (e.g nginx or haproxy). Use docker to run your crap.
Any software engineer should be well capable of setting up a secure server. It really is simple.
The whole process takes maybe 30-60 minutes to setup for someone completely new and following guides.
Render's is simple, true, you just pay $300 for 1TB of bandwidth.
It's crazy how much Merchants of Complexity fooled devs into thinking that running your own server is complicated and you need to pay 1000x to save few minutes of your time.
I get that, a lot. It's the "No True Scotsman" of tech. This is also used as a way of validating college-style leetcode.
Let me introduce you to my GH Activity Graph[0]. See all that green? That's pretty much all coding in Swift; mostly in shipping apps and whatnot. There's a bit of PHP, for server-side stuff, but I like to spend a lot of time, coding frontend app stuff.
Every minute I spend, being a Linux admin, is a minute that I don't spend on executable code. I know that there's a number of folks, hereabout, that can code circles, around me, in Swift, and a few more, that can code circles around me, in PHP, but pretty much all of you, can run circles around me, in Linux admin. I'm not especially interested in competing, there; especially since a number of you are likely the folks that would Do Bad Things to my server, given an opening.
[0] https://github.com/ChrisMarshallNY#github-stuff
You don't have to be a linux admin to be able to setup a properly secure server, nowadays it's quite trivial. I bet you, that if you tried, you would be able to do so in less than an hour (at a relaxed pace).
We get lied to about the complexity of hosting by the cloud. I was able to host forums and game servers on VPSes when i was a young teenager, and I'm not technically gifted, it just wasn't complicated at all.
I tend to like shipping stuff, which means taking Responsibility for its operation, maintenance and security.
That often means a lot of "not fun." My servers are working servers. They have data and capabilities that are important to a lot of "not-Chris" people. It's my job to make sure that they get what they are [not] paying for (I write free stuff). That can be a bit stressful, at times; especially when some bad actor is making life miserable for me. I'd much rather that be someone else's problem, where I can write an email that says "Make it so, Numbah One!", instead of spending two days, wrestling with config files, and CLIs.
People are getting hacked a lot because of this, and docker doesn't seem to care all that much.
Firewalls are made for two things:
So, if your case and if you want to prevent remote access to your database, you have a bad way: create a firewall rule to drop connections to tcp/3306And you have a good way: configure your sql to bind to ::1
The firewall way requires two configuration (hence: complexity) and hide your intent : the mysql say : "I accept connections from everybody", and then the firewall say "I deny all connections".
While the good way is clear and sane : one component who say : "I only accept connections from localhost"
I guess you gain some security, when you don't have to worry about some things. But you also lose some security, because of the added complexity.
sure it still needs work but much much less everyday
What a silly quote. “No other way” except all the other ones. Everyone watching the man and the cat will learn the same lesson. Everyone who hears the story will learn the same lesson.
I never held a cat by the tail, nor have i ever seen anyone foolish enough to attempt it, yet I am certain I know what happens next.
People FAFO when they should know better all the time.
I think the person in the original article was, at least for the sound file.
> I need the whole enchilada: DB, Web Server, Dynamic Languages, etc. Also, for a shipping, production application, with hundreds of users; where privacy and security are of paramount importance.
It seems quite simple to me, you either learn security, or pay for the expertise of someone that has. You need to decide whether it's worth your time.
I would suggest one thing, though - even with the likes of <big cloud>, they will only provide security in limited cases, i.e. DDoS. Nobody at <big cloud> is going to make sure your application logic works correctly - they clearly won't even make sure you use their resources within sensible bounds.
And people wonder why there's only half a dozen websites any more and the "long tail" has vanished. You can post 3.44MB of audio to Facebook or Twitter and know that you'll never be billed for it.
What brought it closer to home was the characteristics of the recently affect site (same number of daily visits, not popular (a bit niche), etc) where similar to mine.
Moving is easy enough, mostly requiring a DNS update, since I prefer to build the HTML locally and then just dumping it somewhere.
What struck me as odd, when looking for an alternative is that almost none of the popular solutions Netlify, Vercel, Cloudfare (AFAIK) actually implement a spend limit.
This seems like such a basic thing to do...
- firefox - userAgent spoofing claiming I'm chrome - pihole
If I change enough of these, I get through the infinite loop. Does not make me happy.
https://vercel.com/blog/introducing-spend-management-realtim...
Based on the info in this thread, Vercel is the only company I'd leave for Netlify.
It’s not really the serverless part that is the problem but rather inability to limit it
eg cloudflare has serverless stuff that you can hard limit
As a small user and not a large organisation, I'll never be able to read the full documentation on throttling every nook and cranny, I want to set a fixed spending limit that if reached, will kill the whole thing, even lose data if it must. But they won't do that because it hurts their bottom line if people are careful with budgets, and sometimes it's tricky to compute billing ahead of time.
A more accurate allegory would be if you told the mechanic to do whatever anyone who knows the registration plate asked, and he did so.
It also has a pretty nicely growing community where people are contributing new templates (I added one for Syncthing) and helping each other debug.
The only issue I've had with it is things kinda fall over if you run out of disk space, which happened when I was running on an instance with just 10GB storage. So a little better alerting or prevention around that would be great but otherwise it has been pretty solid.
https://github.com/coollabsio/coolify
Presumably the costs for a cloud provider are passed on to the cloud provider's customers from that cloud provider's operating costs (with profit). When a spike happens due to, say, a DDOS, does the cloud provider first have to negotiate with its downstream suppliers (e.g., energy, network peering, etc.) to waive the cost?
...
How far down the chain does this go? Or maybe insurance steps in at some point -- or every point -- to break the chain?
Oh and it's not like you can jump into the console to debug.
I gave up after not even vercel could figure it out and merely told me that I should replace said package.
Not that serverless is perfect.
I have seen AWS Lambda generate large bills even with the 1000 concurrent executions limit, but with the help of upstream services. My memory of it is hazy, but I think over a weekend, a function with some dodgy retry logic managed to generate something like 21TB of CloudWatch logs.