Going into a public Cloud has massive advantages. The problem is, most people simply use it wrong...
- They don't right-size their instances: AWS provides a service for this
- They don't using scaling correctly - they just stick to static instances
- They don't scale VERTICALLY, as well as HORIZONTALLY, during a scaling event
- They don't agree to a minimum term or pay anything up front even though they know they will need it, they just stick to on-demand pricing
- They don't use the whole suite of services to build their architecture: it's just EC2 and ALB and off they go
- They use `AdministratorAccess` for all IAM Users and you end up like Uber
I love the idea of hosting on cheap, physical tin - look at the prices Hetzer can get you - but the goodies you get from Cloud cannot be ignored. The automation around provisioning resources alone is simply too powerful (especially when you combine it all with Terraform.)
Then there's the ability to pay for resources up front and STILL be able to "chop" up what you paid for into different units of compute. You can spend $5k/year on a big, fat server and then split it into 15 smaller servers if you like. Same price. No surprises on cost at all.
Egress traffic is the biggest cost, by far, but that can be architected around.
In the case of the article, though, it sounds like they should have gone bare metal to begin with because location is basically irrelevant to them (another example using Cloud wrong.)
I have a feeling people feel "cool" for knowing how to combine AWS services and pretend it's the modern magical way, in fact they're just feeding themselves to AWS by not having radical thoughts.
One instance with some knowledge running your own pieces can do quite some wonders.
But young kids compete by how many buzzwords they know.
You see, you've made quite the fool of yourself here.
I actually run my own hardware on a 250mbit fibre line from my garage. I have several systems in there and know what "running you own pieces" can do for you.
I'm also close to 40 years old. I'm probably older than you are.
I would have gladly engaged you and offered you what I know, but you've chosen to attack rather than work with.
In a past life, I led the migration from datacenter-hosted infra to AWS, and the net result was that we were able to actually increase the size of our infra to support full geo-redundancy, but pay ~40% less to do so.
Our app usage had a massive range of requests per second from single digits during off hours to many hundreds during peak hours. By modeling our usage, we were able to combine reserved capacity with appropriate scaling to manage our costs. Since our customers were primarily on mobile, we'd already invested heavily in minimizing egress traffic. Finally, making full use of many of the management tools aved labor costs, so a smaller operations team could support the business.
There are over a hundred applications offered through AWS many of which are quite clever and offer great help to service providers. Most of these services are also already approved for use with government services which can be a huge boost.
Have to a agree, though, that many who choose AWS do not seem to rigorously evaluate other options like local servers, hosted servers, or alternative cloud services.
It's like "discovering" you reduced your food costs by 80% by going to the supermarket instead of ordering uber eats at every meal... is anyone surprised ?
I’m sure this is well thought out and is probably a good template for other organizations that have reached the point of scale where cost optimization of the workload is needed.
But, the first step I would take to control AWS costs is to complain to your account manager. Internet egress private discounts in AWS are extreme and an open secret. I’d start there before introducing infra tech debt.
I couldn't find mention of comparative hardware/license/support costs for the on-prem servers. Is there that kind of resource out there readily available?
It depends on the server but if they're self-hosting they could be looking at ~$100k per server if they go towards the high end or as little as $3-$10k/server for low or mid grade hardware. They mentioned 300 servers, so I would assume that there was somewhere between a $500k-$5m upfront outlay for their hardware which would need to be refreshed every 3-5 years, which is a huge range but since they didn't disclose specs it is hard to correctly estimate.
For instance, many of us could probably put together a decent cluster of used circa 2012-2017 servers in a rack for $5-$10 grand (not counting storage) which would be enough to run any small or startup business needs for thousands of users.
Also, since this text was IMHO from a dev's perspective and not from a sysadmins, there's a good chance that it could have been individual docker containers inside of VMs running on less physical server. In that case, a single $100k 16x blade server with a few hundred terabytes of storage would probably be more than enough to start and provide a good base line for expansion, so reading back over it I would guess they spent about $250k in hardware to offline from AWS
18 comments
[ 5.1 ms ] story [ 57.6 ms ] thread- They don't right-size their instances: AWS provides a service for this
- They don't using scaling correctly - they just stick to static instances
- They don't scale VERTICALLY, as well as HORIZONTALLY, during a scaling event
- They don't agree to a minimum term or pay anything up front even though they know they will need it, they just stick to on-demand pricing
- They don't use the whole suite of services to build their architecture: it's just EC2 and ALB and off they go
- They use `AdministratorAccess` for all IAM Users and you end up like Uber
I love the idea of hosting on cheap, physical tin - look at the prices Hetzer can get you - but the goodies you get from Cloud cannot be ignored. The automation around provisioning resources alone is simply too powerful (especially when you combine it all with Terraform.)
Then there's the ability to pay for resources up front and STILL be able to "chop" up what you paid for into different units of compute. You can spend $5k/year on a big, fat server and then split it into 15 smaller servers if you like. Same price. No surprises on cost at all.
Egress traffic is the biggest cost, by far, but that can be architected around.
In the case of the article, though, it sounds like they should have gone bare metal to begin with because location is basically irrelevant to them (another example using Cloud wrong.)
- licenses for EC2 or RDS gets sorted automatically.
- Easy, centralized and automatic patch management of EC2 servers.
- If something is getting outdated, AWS will start sending emails so you can upgrade in some favorable time.
- Simple scaling of space, I don't need to buy SSDs compatible with my server, I will just ask for more space and few minutes later I got it.
One instance with some knowledge running your own pieces can do quite some wonders.
But young kids compete by how many buzzwords they know.
I actually run my own hardware on a 250mbit fibre line from my garage. I have several systems in there and know what "running you own pieces" can do for you.
I'm also close to 40 years old. I'm probably older than you are.
I would have gladly engaged you and offered you what I know, but you've chosen to attack rather than work with.
Good luck.
Our app usage had a massive range of requests per second from single digits during off hours to many hundreds during peak hours. By modeling our usage, we were able to combine reserved capacity with appropriate scaling to manage our costs. Since our customers were primarily on mobile, we'd already invested heavily in minimizing egress traffic. Finally, making full use of many of the management tools aved labor costs, so a smaller operations team could support the business.
Have to a agree, though, that many who choose AWS do not seem to rigorously evaluate other options like local servers, hosted servers, or alternative cloud services.
https://news.ycombinator.com/item?id=33053416
https://news.ycombinator.com/item?id=33007873
https://news.ycombinator.com/item?id=33036448
But, the first step I would take to control AWS costs is to complain to your account manager. Internet egress private discounts in AWS are extreme and an open secret. I’d start there before introducing infra tech debt.
For instance, many of us could probably put together a decent cluster of used circa 2012-2017 servers in a rack for $5-$10 grand (not counting storage) which would be enough to run any small or startup business needs for thousands of users.
Also, since this text was IMHO from a dev's perspective and not from a sysadmins, there's a good chance that it could have been individual docker containers inside of VMs running on less physical server. In that case, a single $100k 16x blade server with a few hundred terabytes of storage would probably be more than enough to start and provide a good base line for expansion, so reading back over it I would guess they spent about $250k in hardware to offline from AWS