Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
I made a mistake and accidentally created a serverless function that called itself. In a recursive loop, with a 30s timeout. I thought I fixed it and deployed the code to the dev environment.
I have had an AWS Billing alert (Budgets) set up to prompt me when my monthly budget goes over $300 (my usual bill is $200/month).
Imagine the terror when I woke up the next day to see the AWS Billing alert email saying I already owed $1,484! I removed a function and deployed it again in 30 minutes, but it was too late. It has already run for 24 hours, using over 70 million Gb-Second!
Only after that I've learned that AWS Billing alerts do not work this way for CloudFront. You get delayed information on charges because they collect them from all regions.
On the following day, the bill settled at a shocking $4600. This is more than we have ever spent on AWS all time.
CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront).
Now, I understand that I should have created CloudWatch alarms, which would alert me when the number of requests exceeds the limit. The problem is, that they need to be set up per region, and I got CloudFront charges from all points of presence.
I am a big proponent of the serverless approach. It makes it easy to scale and develop things (e.g., you get PR review version branches for free, both frontend and backend code like Vercel does). But now, I am unsure because such unexpected charges can ruin a side-project or emerging startup.
Now I am waiting on a response from AWS Support on these charges; maybe they can help me waive part of that.
What is your experience with it? Would you recommend to use to build a new product if you are bootstrapped, 3-person startup?
347 comments
[ 3.2 ms ] story [ 264 ms ] threadThanks, already done that, and waiting for it to be reviewed by AWS. Support was very responsive, though.
I racked up a $8k AWS bill for my university when I was leading a club. A few emails to AWS support and it was all resolved. Although there might've been more leniency since I was a student.
I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night.
In the morning, when I learned what had happened, my bill was over $3k.
I fixed the issue and emailed AWS asking for some relief, and they called me and let me know they were waving all the charges.
So, perhaps you too can beg for mercy?
Hypothetically, the contract could say Jeff Bezos will come to your house and personally kill you, but there's no consentual murder in most places
Now, Amazon would be entirely within their rights to cancel your account and refuse to do business with you after this, but they would not have the right to collect that money from you, or to keep that money had it already been charged to you.
Your only legal recourse under Title 18 Code 1030 is against the "violator". Amazon did not violate your computer systems and commit these offenses.
> Any person who suffers damage or loss by reason of a violation of this section may maintain a civil action against the violator to obtain compensatory damages and injunctive relief or other equitable relief.
On that basis, your contract stipulates who is responsible for fees associated with use of your AWS key by "any other third party".
> You are responsible for all applicable fees associated with use of the Services in connection with IAM, including fees incurred as a result of any User Credentials. You are responsible for maintaining the secrecy and security of the User Credentials (other than any key that we expressly permit you to use publicly). You are solely responsible, and we have no liability, for any activities that occur under the User Credentials, regardless of whether such activities are undertaken by you, your employees, agents, subcontractors or customers, or any other third party. You are responsible for the creation, distribution, and security (including enabling of access) of all User Credentials created under your AWS account, including credentials that you have used IAM to create or disclose to other parties.
The computer fraud in this case was not committed against you. It was committed against Amazon. Amazon grants you access to their services, the account does not belong to you. The damages here are not made against you, they are made against Amazon.
Just like in my example, the violator committed fraud against the "buyer" of the car. Neither Amazon or the "buyer" have recourse against you for the supposed owed property/bill, they have to extract damages from the violator. You are not responsible.
On your second point, I will repeat myself: it doesn't matter what the terms or contract say. Such agreements commonly hold terms that are in direct opposition to US law and have no legal basis. Their entire purpose is to dissuade you from pursuing your legal rights at a cost to the company.
This unfortunately isn't true. It also sounds like he created an app key from his root account that enabled anyone to literally impersonate him.
A typical use case is to create a user that has only the specific rights that are needed and generate an app key for that user. For example, I have a user that can only read S3 buckets. If it were to leak, the worst that would happen is I would leak some encrypted backup data.
In a situation like this, Lambda is almost pure profit. Their actual spend here was negligable.
They are almost certain to waive the fee, because they don't want the perception among developers that AWS is a time bomb.
After a surprise bill like this, I would re-evaluate what serverless is actually giving me.
> We compared the three-year total cost of ownership of a VPS, such as a DigitalOcean Droplet, against two equivalent leased or purchased bare metal servers. We estimated that the leased option costs about half as much compared to equal resources in the cloud, and owning the servers would cost less than a quarter of the pure cloud options.
[1] https://freebsdfoundation.org/our-work/journal/
[1] https://freebsdfoundation.org/wp-content/uploads/2022/06/Jou...
Everything on aws is a clusterfuck designed to suck money out of enterprise businesses
For example, it's totally understandable that the alarms can be specified per region, why shouldn't it be like this?
Also the global AWS billing $300 alert seems to have worked but you were asleep as far as I understand. If it was a call-out style alert, then you would've noticed in the middle of the night and could've stopped it.
The only thing I agree is frustrating is this: > CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront).
Maybe you can argue that you weren't made aware of this but idk... keep us updated
Perhaps AWS should have "personal/developer" accounts that have this enabled by default and continually warn you about it, whereas "company/enterprise" don't have them.
Ultimately whatever solution you put in place, someone is going to complain about it. At least with the system they currently have in place they can reimburse customers. Whereas it is a lot harder to fix their reputation after they've automatically stopped production services.
Ive seen nobody on HN, twitter, reddit complain about "my site was down during heavy business since i turned on hard billing setting". Not a single person.
However, I see frantic after frantic post of "I was testing something on AWS and it caused me a $X000 or $X0000 bill."
But as the posts in here are apt to suggest - you can always beg AWS support for a reversal. Great plan there.
(Also one could make the "nobody uses Azure" joke here.)
Personally I think that much of AWS is "way overpowered" for the normal person/business, and you shouldn't be playing with it if a $X0k bill would be impactful (as likely other solutions are much better tuned to your needs and money).
When you are handed a tool that has multiple hidden guns and explosives inside of it, and ends up blowing your foot off is malfeasance of the people who handed the tool to you.
AWS is that tool. And given that Azure can implement these guard-rails and AWS chooses not to tells me all I need to know.
> Personally I think that much of AWS is "way overpowered" for the normal person/business, and you shouldn't be playing with it if a $X0k bill would be impactful (as likely other solutions are much better tuned to your needs and money).
Please compare and contrast this with "Learn AWS for furthering your career".
Yeah, but I figure as long as Amazon doesn't immediately remove stored data, the damage of the footgun would be minimal. Speaking for myself, of course, I'd rather have a short outage than an unexpected thousand dollar overnight expense. It seems so trivial that it's unclear why AWS would not implement this feature. The only explanation that makes sense is that they want these surprise bills to occur.
Given how easily they reverse the bills, I suspect that they have a policy of doing it (perhaps a few times per account, something to prevent abuse) because they really don't want to trigger the above scenario.
Unless you're a big company, "we stopped your function in the middle of the night" is a whole lot better than "we ran your function all night and you owe us $4k".
AWS already has a recourse for incidents like these: refund the spend. That is far more reliable than trusting an organisation can tolerate an outage.
If that alert triggered earlier (and $300 would have been triggered in an hour), my all accumulated charges would be only $400 not $4500.
So the hard learning here is that CloudFront charges takes time to appear on your bill, up to 24 hours.
It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss or in unpredictable ways, but in most cases that would be preferable for these users as opposed to a couple orders of magnitude increase in billing costs.
But the cloud providers don't support hard billing limits because they like people fucking up and accidentally running up their bill. After all, it's probably only a small fraction of users that go through all the humiliating rigamarole of unwinding a provisioning mistake.
So yeah, good on Amazon for being so generous with the band aids, but maybe they should try a little harder at helping their users not shoot off their toes...
former AWS SDE here
I don't believe it would be "fairly simple" to build a completely new off switch into 150+ services, likely with multiple integration points in each service. In addition, the mere existence of an off switch introduces new failure points, where failure directly turns into downtime.
The effort to implement this is far from trivial, removes resources from implementing other features that the really large accounts are asking for, and adds complexity with direct availability risks. It's not at all surprising they don't implement this.
It's got 0 to do with why the feature doesnt exist though.
They wrote it off ~6hrs after we filed a support ticket about it.
I really want to use Lambda for public endpoints but it just scares me.
That said it does take a level of awareness to set these things up. Some of what we try to do at SST is turn on these for you automatically so you're not being punished for not knowing something
It could not have been easier to get AWS to remove the charge. A quick email to support with a brief explanation and it was immediately accepted. The hardest part was that they wanted a very specific request for how much I was asking to be refunded. So I had to go back and calculate my average costs per service and compare that to the charged costs. After that it was immediately refunded.
They aren't just handing these things out though. They made me read and acknowledge I had read their service agreements and basically swear that I know what happened and it won't happen again. Really painless process overall, all things considered.
Fortunately, AWS was kind enough to reverse the billing. Had this been Google Cloud, I would've gotten the cold shoulder and a low key threat that if I reverse the transaction I would lose access to my other paid Google products outside GCP :/
S3 has some setting where you can log activity on a bucket into another bucket
But that setting allows you to set the destination bucket to be the same bucket that you're monitoring. So ~30s after something happens on the monitored bucket, S3 writes a log into the same bucket. And then that activity triggers the logging again. So every ~30-60s, forever, there's a little log written into the bucket.
It takes a while to add up to something noticeable if your monthly AWS bill is already a few digits long. It's super fun to sift through the bucket a few months later when you're trying to figure out if there's any real data in the bucket or just endless logs.
After I explained my situation (a $5k bill for an inactive side/toy project is extremely painful), and I provided great details about what I think happened to cause it, they wrote off the charges.
> my usual bill is $200/month
How many req/second are you serving? What kind of things are happening?
It seems like the bills are outrageously expensive when it comes to various cloud services, as I'm personally hosting a service that does between 10-100 req/second on average during a month, and my monthly bill end up being closer to $40/month, including traffic and everything. I'm running a database on the same server, and 20% of the requests writes stuff both to disk and to the database, while the rest just reads from DB or disk.
The whole setup took around 5 hours to setup on one day, and have been running flawlessly from day one, we haven't had to migrate servers yet after ~6 months of production usage. Probably one day we're gonna have to upgrade the server to a $80/month one, but that's a one-time thing and our revenue easily covers that.
I know I keep harping on about this, but [if you have a vague idea what you're doing] you can squeeze an awful lot out of a cheap VM.
I'm currently working on a project that we're deliberately prototyping using ultra-cheap VMs. If it takes off we know how to scale it up, if it doesn't the costs stay very very low.
but now, being a 1 person dev team, it is challenging in maintenance.
My fear is, being on a vacation, and suddenly this VM dies. It might take too much time to bring it back online, and I might be out of good network coverage.
took about a day to rewrite and a week of 30 mins per day to figure out how to optimise my code for self healing.
knowing my bill is capped by number of VMs * memory, saves a lot of stress
We, developers, never liked the reboots though, always wanted to find out root cause, so that we won't be paged again. So, I guess, we moved to the cloud. Now we don't get paged in the middle of night.
But yeah if it was my own company and I was on vacation, I rather take 30 seconds to reboot the server instead of worry about paying thousands of dollars.
As for lack of network coverage, you could do scheduled daily/hourly reboots while on vacation, if it makes sense for your service. Or if your an outage will cause massive disruption to your users, then perhaps hire a part-time sysadmin while on a vacation.
With a bit of refactoring and simplifying it tended to cut out a lot of issues. I suppose the issue is that a lot of people don’t really know what to look for or how to anticipate issues in complex infrastructure (which is totally fair, I only learned via trial by fire and have done some really stupid stuff).
I don’t think I ever encountered a case where the code was correct and bills were too high. Some AWS/GCP configurations would be pretty bad, but the code would also tend to be incredibly inefficient.
I always encourage people to respect people who are great at dev ops and to either hire them if they’re big enough or just consult them if they’re uncertain about things. Throwing a day of consulting rate at someone smarter than you is a great way to learn and it could easily save you money in the not-so-long term.
I’d say that because they’re crazy to rely on me for infrastructure, but I’d make things better if I was around already and they asked me to help out. But I’m no substitute for someone who actually knows what they’re doing.
For comparison, I've been running a paid Slack app (a few $K per year) that manages to run entirely within the free tier on Google Cloud Platform.
In any case the free tier will definitely allow you to handle enough traffic to figure out if you've got a viable business model, which I guess is the point.
These recursive requests were going at the rate 17k/second with 30s timeout each.
The benefits of the current approach is I don't need to manage any servers, and I have different environments for free. Also sizing is not an issue, I just tune AWS Lambda limits to be able to serve one single request.
I will need to invest some time to understand how big the instance should be (how much memory) because I struggle to measure how big it should be without going out of memory or CPU.
- After crossing a certain threshold in scaling needs, Lambda costs more than regular EC2 on ELB
- Lambda cold-start times can be a deal-breaker when users first visit your website. If you contact AWS they will tell you to setup a simple cron job that keep lambdas "warm". But AWS provides no visibility in what's warm or cold, or which endpoints link to which lambdas.
- Dealing with Cloudwatch logs of various lambda runs (IMHO) is objectively a bad dev experience. Query insights is getting better, but is still a pain to work with.
- To reduce deployment and development times, you'll eventually want to deep-dive into lambda layers. Modern problems modern solutions.
- One lambda calling and awaiting another lambda is not a supported first-class use-case. There's no API that allows you to get the status of a lambda run. There's a hack around this where you use AWS Step-Functions. Modern problems modern solutions.
We're still on AWS full-stack "serverless" for our webserver and realtime stream processor. At the time I didn't know what I was getting my company into. I wish I just made a Flask webserver instead.
... have you considered automating this? Alarms are pretty straight forward across all cloud platforms. Since you're using AWS: CloudWatch has anomaly detection. I haven't used it personally but perhaps it's worthwhile to look into: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
Soon as I saw this headline, I was completely shocked and dropped everything to check my usage. Luckily I had 'cdk destroy' the big projects, but I had dozens and dozens of lambda and S3 buckets it failed to clean up. I spent probably 1 hour clicking through the web interface deleting them in fear of what happen to OP
I think I'll just stick to LocalStack and VPS.... actually my datacenter friend said I could bring up my deep learning station to this DC might just abandon AWS idea.
Sorry that happened, always one of the scarier parts of using AWS. This sounds like an especially tricky one with the standard billing alerts not even catching it.
I already contacted them, my past experience with AWS was pleasant, it is just this CloudFront delayed billing should be better clarified in the docs I suppose.
The technical details is outbound requests is given a role encoded in the user-agent, and then I can easily filter out incoming requests by user-agents [1].
[1] https://observablehq.com/@endpointservices/webcode-docs#opti... (see loop prevention flags)
1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck for the benefit of learning that experience and getting your hands dirty with AWS.
2. AWS billing is actually really good for what it is. If you've ever run any non-trivial operational system (in the real world), you would know how hard it is to collate all expenses and get them tallied up. AWS collates all billing data with ~24h lag and you can slice and splice it to your heart's content. After all it's a complicated distributed system that they've managed to build that doesn't slow down your services or otherwise get in the way!
And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be.
> AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills.
If there were hard limits, there'd be no need to beg AWS support for leniency, which they can capriciously choose you don't deserve.
https://azure.microsoft.com/support/legal/offer-details/
Larger companies see $4k as a nothing; pay and move on. Household budgets, not so much.
In real life when you hit your card's limit, your transactions get declined. Straight away.
I had this last month in a supermarket after my (personal) checking account didn't have enough money to cover my purchase, I'd completely forgotten to transfer money from my business account.
My bank wasn't prepared to let my account go overdrawn, not even by the equivalent of $20, which is absolutely their right.
Amazon, OTOH, benefits in lots of ways by not implementing this mechanism.
You put stuff in your cart.
You go through the register. You agree to "prevailing price".
No prices show up because they are "calculating".
You leave the store.
A day later, you're hit with a surprise bill that's 10-100x more than you though. A $100 transaction ends up being $1000-$10000.
There's no refunds.
The dispute procedure is to beg and hope they "let you ignore it".
Except for when it doesn't. I've got two primary current accounts, one with a "legacy" bank in the UK and one with a modern bank. The legacy bank is happy to let me go into an unplanned overdraft, and charge me for the privilege of doing so.
"Been there, done that". Not there any more!
What is true--not necessarily in order is:
- I suspect AWS in aggregate probably makes a fair bit of money on overages that a user eats but would have had a hard circuit in place if they could have, and
- Even reasonably designed hard circuit breakers (e.g. we cut off access to your stateful data unless you pay your bill but we won't delete it for 30 days) are still giving developers a potentially well-hidden foot-gun for a production environment that management might not actually want.
At this moment, any dev with AWS keys has an unlimited month-per-month credit line that the company is on the hook for paying. And at best is the hope and prayer that the billing notifications aren't utter shit.
The truth is that developers sometimes require expensive things to get their job done efficiently. The other truth is that we often vastly overestimate the SKU. And we also leave things running.
The storage bill alone exceeded that at four startups I've worked with. When your job is to manage X PB of data with Y TB arriving daily, you're fairly constrained on the cost floor of your operations.
> You are making bad engineering decisions if you go through that in under a year.
Fine. You lack the imagination to conceive of a use case, or you lack the knowledge and skills to get the job doing it. Either way, this comment is rude and ignorant of the diversity of software challenges that exists in the real world.
I've never run up a huge AWS bill accidentally, but I personally know 2 people who have, and neither was refunded, even after asking. In both cases we are talking $400-$800, enough to really hurt someone, but not bankrupt them.
I wouldn't bet my bank account on that always holding true. If it's not in the terms and conditions that they'll refund you for accidental mistakes that lead to high billing, then you're gambling if you assume they will.
I once moved a project from AWS to Digitalocean for a small team (AWS was just all they knew, so it was what they used) and I was able to cut the monthly bill down quite a bit.
It isn’t that DO is inherently cheaper or better. It’s just dead simple so it’s easy to deploy with only what you actually need, with easy limits and visibility on what gets spun up. In some cases it’s arguably less cost efficient, but it’s really hard to mess up.
For the team I was supporting, simply having the visibility and a simple tool was worth a lot in saved time. They previously spent way too much time on AWS, and couldn’t even get the right infrastructure with the time they invested.
So, maybe something worth considering at least. Good luck with the bill! You’re certainly not alone (I got an $800 charge for a db I forgot to kill a few years ago).