Hi,thinking of creating an app to limit AWS account spend, need honest opinions
Long story short, I recently got an aws account for side projects but I am scared that someday someone's going to go crazy with one of my public websites and I'd end up with a large $XXX bill. I have also heard horror stories from this sub about people trying something or forgetting to turn resources off and ending up with a large budget. I know we can set budgets and all but it doesn't work for people who want to cap the total $ spent.
I am a little surprised aws hasen't introduced this feature despite many requests (although when I think from a business pov, it makes sense).
Basically I am want to create an app where you could say something like
"I ONLY want to spend $XX per month, if my bill is more than that, shutdown my resources"
My idea is to create a website where you can set a budget amount (monthly?) for your account and setup rules for different resources ( like shutdown EC2, make S3 bucket private etc)
I am pretty good with creating backend tools and worked a lot with aws lambdas and s3.
I'd love to get some ideas from you guys.
Also if you don't think its gonna be helpful, let me know that too.
thanks
15 comments
[ 3.1 ms ] story [ 48.1 ms ] threadSo you will have the option to auto stop services to minimize spend or terminate services if stopping will not reduce the bill (rds is an example).
User probably needs to be able to set priorities, locks, etc to prevent disruption of services in case they are using it for production.
The biggest question is "where are you going to get your data about current spend?. AFAIK the budget data lags behind usage (which is what I assume is the reason people haven't done this before). You can do this at a per-service level (e.g. EC2) by periodically polling EC2 for active resources and keeping a running tally of cost by yourself. This works well for some services like EC2, RDS (although serverless Aurora gets tricky), etc. However for other managed services like S3 or Lambda, it's harder to keep track of how much you have spent.
Cloudtrail could be useful here, but it probably doesn't have enough information to calculate actual cost (e.g. Lambda charging based on invocation time or Athena charging based on the volume of data scanned).
But even something as simple as a tool that keeps track of just your EC2 instance spend and shuts them down if it gets too high would be useful to developers who have AWS accounts for their side projects.
> The biggest question is "where are you going to get your data about current spend?. AFAIK the budget data lags behind usage
interesting, I was thinking I could poll billing metrics (total charges etc) to figure out account level charges.
In your experience, do you care more about EC2 costs or that was just an example?
If it were that easy, this would already exist ¯\_(ツ)_/¯
I think many people use AWS primarily for s3 and EC2 (and s3 costs are negligible for most side projects) so if you were picking a single service, EC2 would probably be the most broadly applicable (and IMO the most realistic one to actually build). On the other hand I would imagine that for many side project developers, their EC2 costs are the easiest to control - even if you do autoscaling, you can cap your max nodes.
What is most useful is tracking the total AWS spend because what really gets you is the service you weren't expecting to cost as much as it did, but that is a very very very hard problem.
after thinking about your comment on EC2, I think this tool could help smaller teams where you don't know who's doing what with the resources and you just need an upper limit of your monthly/daily spend.
about the last point, yes it's very hard to figure out. Also I thought most people check out the pricing page before trying out any aws services, wouldn't that give an estimate of what they are getting into (i.e. RDS, API gateway) ?
But again I always forget about the data transfer charges so you might be right about this.
> I think this tool could help smaller teams where you don't know who's doing what with the resources and you just need an upper limit of your monthly/daily spend.
Yeah, definitely!
> But again I always forget about the data transfer charges
Yeah. AWS can have some pretty complex cost models since they charge based on the amount of resources you actually used - there are way more axes of cost than most people have time to think through.
If you end up building this, I suggest considering that many people will not want their resources automatically shut down, either because they want to serve the spike in traffic or because they are paranoid that there could be a mistake and have production infrastructure accidentally shut down. I would consider designing it so that users can choose to get a Slack message or something instead of having the resources shut down.
It's entirely unnecessary to be exact, you need to be close enough. This is especially true since it's a feature a lot of people want, they'll accept some weakness in the system within reason.
You could offer a flex range the customer can set, a tolerance. If you can't reliably get near that (eg if you can't get within N% when back-checking your estimates versus what ends up being the actual billing), then you let the customer know with a status update and the customer can make a decision as to what to do at that point (they could stop using your service, they could adjust the tolerance, and so on).
Require a minimum number of polled data points before the system comes alive and becomes capable of shutting down services. You need a certain number of records before you can reasonably extrapolate anything.
This system won't work well in cases of normal usage with a sudden extreme burst that blows out the customer's budget (where the customer is hoping to prevent that type of unexpected one-off surge). You could do an accelerated estimate guess if you see a very large, very unusual spike, and jump ahead as a means to over-protect (another setting the customer could choose, they could set a caution level that guides how much you should accelerate ahead in cases of huge surges).
If you're successful, Amazon will kill you or buy you (possibly ask to buy you for cheap, then kill you if you refuse). As with similar hyper platforms, the platform eats its own ecosystem inhabitants in the name of growth. And this is a blatantly obvious feature that is widely demanded, so if you figure out a way to do it even modestly well, they'll knock you out eventually. Print money for a while if you can, charge as much as you can, prepare for the inevitable; consider selling to a cloud service company if you can find the right offer.
Also what do you think about AWS Cost Explorer API? It looks like it can provide monthly/daily charges and we can build rules for customer tolerance, like you mentioned.
Also, I don’t want my production account to be shut down. Notified, sure. But if my spending takes off because Elon Musk tweeted about my website and I’m getting a lot of hits (and hopefully a lot of new paying customers) I definitely do not want my service to go down.
In other words, your idea isn’t for me, but it might well work for side projects where you don’t want to spend too much and don’t care about the service going down. In any case the security aspect is going to be tough to manage properly, and is going to make your service a juicy target for hackers.
This is already an AWS feature: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...