Ask HN: How can I quickly trim my AWS bill?
Hi HN,
I work with a company that has a few GPU-intensive ML models. Over the past few weeks, growth has accelerated, and with that costs have skyrocketed. AWS cost is about 80% of revenue, and the company is now almost out of runway.
There is likely a lot of low hanging cost-saving-fruit to be reaped, just not enough people to do it. We would love any pointers to anyone who specializes in the area of cost optimization. Blogs, individuals, consultants, or magicians are all welcome.
Thank you!
133 comments
[ 3.2 ms ] story [ 194 ms ] threadI have loud and angry thoughts about this; https://www.lastweekinaws.com/blog/ has a bunch of pieces, some of which may be more relevant than others. The slightly-more-serious corporate side of the house is at https://www.duckbillgroup.com/blog/, if you can stomach a slight decline in platypus.
What exactly are we talking about here?
Couldn’t you build a dual NVIDIA 20XX / 32 core / 64 GB for a sub $5k and then save money while training/developing faster?
Are you spending on bandwidth? See if there's compression you can enable. Ec2? Can your reduce instance sizes or autoscale down instances you're not using overnight? Elasticache or elastic search? Tune your usage, store smaller keys or expire things out.
https://github.com/similarweb/finala seems promising
You don't provide a lot of detail but I imagine at this point you need to get "creative" and move at least some aspect of your operation out of AWS. Some variation of:
- Buy some hardware and host it at home/office/etc.
- Buy some hardware and put it in a colocation facility.
- Buy a lot of hardware and put it in a few places.
Etc.
Cash and accounting is another problem. Hardware manufacturers offer financing (leasing). Third party finance companies offer lines of credit, special leasing, etc. Even paying cash outright can (in certain cases) be beneficial from a tax standpoint. If you're in the US there's even the best of both worlds: a Section 179 deduction on a lease!
https://www.section179.org/section_179_leases/
You don't even need to get dirty. Last I checked it was pretty easy to get financing from Dell, pay next to nothing to get started, and have hardware shipped directly to a co-location facility. Remote hands rack and configure it for you. You get a notification with a system to log into just like an AWS instance. All in at a fraction of the cost. The dreaded (actually very rare) hardware failure? That's what the warranty is for. Dell will dispatch people to the facility and replace XYZ as needed. You never need to physically touch anything.
A little more complicated than creating an AWS account with a credit card number? Of course. More management? Slightly. But at the end of the day it's a fraction of the total cost and probably even advantageous from a taxation standpoint.
AWS and public clouds really shine in some use cases and absolutely suck at others (as in suck the cash right out of your pockets).
And when they aren’t always the best. It’s often because you don’t know what you’re doing.
It’s too uncommon for people to over provision. Or go with too many services when they don’t need to.
Like let’s have a database and cache service and search search. When 95% of the time they only need the database because it can do full text searching adequate enough and they don’t have the traffic to warrant caching in redis, and can do basic caching.
They don’t take advantage of auto scale groups, or run instances that are over provisioned 24/7.
I’ve seen database instances where when it’s slow they throw more hardware at it instead of optimising the queries and analysing / adding indexes.
The biggest cost of cloud providers is outbound data. The rest is almost always the problem of the Developers.
Do we know if the instances used for MLing are running 24/7 idle until customers use them? Do we know if the utilisation is optimal for the workloads?
We know nothing. So claiming that cloud providers are not good is very far from the problem and not helpful.
The statement is not that AWS is "not good". The statement is that AWS is very expensive, specially for computational tasks, and there are cheaper alternatives around.
AWS is notorious for positioning their services as a way to convert capex into opex, specially if your scenario involves a SaaS that might experience unexpected growth and must be globally available. Training ML models has zero to do with those usecases. It makes no sense to mindlessly defend AWS as being the absolute best service around for a job it was not designed for and with a pricing model that capitalizes on added value on things that are not applicable.
As I said I have examples of how Developers often cause large bills.
And I explained why we can’t help with the OPs large bill.
You’re saying that with ML there is absolutely 0 way to reduce costs on AWS which is absolute rubbish.
I feel that's where you keep missing the whole point. Somehow you're stuck on thinking that an expensive service is not a problem if you can waste time micromanaging and constantly monitoring expenditures to shave off a bit of cost from the invoice. Yet, somehow you don't register in your universe the fact that there are services out there that are both far cheaper and arguably better for this use case.
Therefore, why do you keep insisting on the idea of wasting time and effort micromanaging a deployment like pets to shave off some trimmings off a huge invoice if all you need to do to cut cost to a fraction of AWS's price tag is to.... switch vendor?
And that for cases like MLing it’s impossible to optimise costs.
Got ya.
No, I'm pointing you the fact that developers are able to do exactly what they want with less work and far cheaper by simply moving away from AWS and picking pretty much any vendor. Why do you have a hard time understanding what others are telling you and understand anything that points that AWS is not the best solution for all usecases, specially those they were not designed for?
"I don't know how to use cloud so cloud is bad"
That is, I am not sure "public cloud, if you spend lots of effort to optimize it and ask devs to be careful, can be as cheap as a naive on-prem implementation where devs don't need to be careful" is an argument for public cloud.
I’m also not suggesting optimising and being careful is an argument for cloud. I’m saying that ruling out cloud is stupid. You can absolute have a Low cost solution perform very well on a cloud provider. The OP seems to think it’s not possible.
A balanced approach is to only put the most expensive hardware portion of the business with the smallest availability requirement in colo, and horizontally scale it over time. Simultaneously use a cloud provider to execute on the cheap stuff fast and reliably.
Go for some colocation facility where costs are predictable.
We really need some more details on your infrastructure, but I assume it's EC2 instance cost that skyrocketed?
A couple of pointers:
- Experiment with different GPU instance types.
- Try Inferentia [1], a dedicated ML chip. Most popular ML frameworks are supported by the Neuron compiler.
Assuming you manage your instances in an auto scaling group (ASG):
- Enable a target tracking scaling policy to reactively scale your fleet. The best scaling metric depends on your inference workload.
- If your workload is predictable (e.g. high traffic during the daytime, low traffic during nighttime), enable predictive scaling. [3]
[1] https://aws.amazon.com/machine-learning/inferentia/
[2] https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-sca...
[3] https://docs.aws.amazon.com/autoscaling/plans/userguide/how-...
Luckily, it's pretty simple to handle interruptions for neural network like models that train over several iterations. Just save the model state periodically so you can shut the instance down whenever the price is too expensive and start training again when the price is lower.
It's unlikely that your users are going to notice the accuracy difference between the linear model and the GPU-intensive one unless you are doing computer vision. If you have small datasets, you might even find the linear model works better.
So it won't affect revenue, but it will cut costs to almost nothing.
Supporting evidence: I just completed this kind of migration for a bay area client (even though I live in Australia). Training (for all customers simultaneously) runs on a single t3.small now, replacing a very large and complicated set up that was there previously.
Eg are there some clients losing you money that you can either let go or raise prices for?
- get devs on GPU laptops
- for always-on, where doable, switch to an 8a - 6p policy, and reserved. Call aws for a discount.
- use g4dn x spot. Check per workload tho, it assumes single vs double.
- consider if can switch to fully on-demand if not already , and hybrid via GCP's attachable GPUs
- make $ more visible to devs. Often individuals just don't get it, too easy to be sloppy.
More probably doable, but increasingly situation dependent
Have you (or anything else here) had experience doing this? Did it end up being a worthwhile approach? (Even for a while)
If training only short while, may do better by setting up a cloud training workflow that only has the server on while training. If on a lot, then a private box makes more sense (ex: lambdalabs, at home/office/colo). Then setup as a shared box for the team.
A lot of time ends up dev, not actual training, and folks end up keeping dev cloud GPUs on accidentally. We still use cloud GPUs for this, but have primary dev on local GPU laptops. For that, we started by System76 for everyone (ubuntu Nvidia), but those had major issues (weight, battery draw...). I then did a lightweight asus zenbook for myself, but that was too lightweight all around. Next time will do more inbetween or explore Thinkpad options.
And yep, as a small team, this mix dropped our cloud opex spend by like 90%, and pretty fast to offset the capex bump.
Lead time might be a problem for you but you can probably do it in a under a month if you take available stock at your vendor. I work with a company called PogoLinux (http://pogolinux.com) out of Seattle and they sell boxes that have 4 GPUs in them.
That said -- the other advice is right. You can probably get by with a much simpler model. The coloc route would probably only be better if you are can't change the models due to people constraints and the ML stuff doesn't have a lot of AWS dependencies. SysAdmins are a lot easier to find and hire than ML specialists.
You don’t have to use cloud services.
For example : Scaleway, OVH, or Hetzner.
Sorry to hear that. I’m sure it’s super stressful, and I hope you pull through. If you can, I’d suggest giving a little more information about your costs / workload to get more help. But, in case you only see yet another guess, mine is below.
If your growth has accelerated yielding massive cost, I assume that means you’re doing inference to serve your models. As suggested by others, there are a few great options if you haven’t already:
- Try spot instances: while you’ll get preempted, you do get a couple minutes to shut down (so for model serving, you just stop accepting requests, finish the ones you’re handling and exit). This is worth 60-90% of compute reduction.
- If you aren’t using the T4 instances, they’re probably the best price/performance for GPU inference. If you’re using a V100 by comparison that’s up to 5-10x more expensive.
- However, your models should be taking advantage of int8 if possible. This alone may let you pack more requests per part. (Another 2x+)
- You could try to do model pruning. This is perhaps the most delicate, but look at things like how people compress models for mobile. It has a similar-ish effect on trying to pack more weights into smaller GPUs, or alternatively you can do a lot simpler model (less weights and less connections also often means a lot less flops).
- But just as much: why do you need a GPU for your models? (Usually it’s to serve a large-ish / expensive model quickly enough). If you’re going to be out of business instead, try cpu inference again on spot instances (like the c5 series). Vectorized inference isn’t bad at all!
If instead this is all about training / the volume of your input data: sample it, change your batch sizes, just don’t re-train, whatever you’ve gotta do.
Remember, your users / customers won’t somehow be happier when you’re out of business in a month. Making all requests suddenly take 3x as long on a cpu or sometimes fail, is better than “always fail, we had to shut down the company”. They’ll understand!
I stopped using gpu's, "Vectorized inference isn’t bad at all!". This soo much, I was blinded with gpu speed, using tensorflow builds with avx optimization is actually pretty fast.
My discovery:
+ Stop expensive GPU's for inference and switch to avx optimized tensorflow builds.
+ Cleaned up the inference pipeline and reduced complexity.
+ Buying compute instance for a year or more provides a discount.
- I never got pruning to work without a significant loss increase.
- Tried spot instances with gpu's that are cheaper. Random kills and spinning up new instances took too long loading my code. The discount is a lot, but I couldn't reliable get it up. Users where getting more timeouts. I bailed and just used cpu inference. The gpu was being underutilized, using cpu only increased the inference to around 2-3 seconds. With the price trade off it was a more simpel,cheaper and easier solution.
However, this morning I was playing around with Scaleway bare metal [1] and General Purpose instances [2] -- I am thinking of making a switch for high CPU tasks.
[1] https://www.scaleway.com/en/bare-metal-servers/
[2] https://www.scaleway.com/en/virtual-instances/general-purpos...
The main point is that physical servers are much cheaper than VMs and provide significantly better performance as well (see my benchmarking and comparison: https://jan.rychter.com/enblog/cloud-server-cpu-performance-...).
Their block storage volumes look interesting, but I couldn't find any information on performance guarantees, or even claims.
Anyone have an idea about performance (IOPS or MB/s)?
I only use their dedicated servers with NVMe SSDs and have never benchmarked the I/O.
I will say that more generally speaking, there has been a lot of recognition in the industry at large that AI-driven startups all face this challenge, where the cost of compute eats up most of the margin. There is no easy solution to that, other than to make product-level decisions about how to add more value with less GPU time.
Other than that, you should also look at your architecture. Often there is opportunity to save there as well.