I've only worked at companies running in the cloud. Is there a console (open-source or otherwise) and other tooling similar to AWS' for managing your resources in your own data center?
I can definitely see advantages for very specific use cases in favor of bare metal colo. But AWS does provide a good amount of value in flexibility and reliability. If you have a whole lot of bandwidth or storage needs you should probably look at a hybrid system. But you will probably pay for it with increased engineering and maintenance costs.
Another interesting thing is that the farther you go from the CPU cache, the more copies of data and the more extra processing you have to do.
Think of how many copies of the data you need (and how much processing it takes) to use:
- a L3 CPU cache.
- a 16GB RAM cache.
- an embedded C/Rust/Go SQLite database.
- a distributed, fault tolerant Cassandra/elastic cache cluster.
You can literally solve the same typeahead/autocomplete/user lookup/session management/etc.. features using the last three of these - but the CPU cycle, memory usage and latency costs (plus complexity) explodes.
I'm always impressed by people that handle 1 million connections on a single box while I'm watching teams trying to handle a thousand connections with a whole fleet of infrastructure and caching layers.
Then again, part of the issue is the wrong language and wrong designs patterns. Highly optimized Rust and Go someone obsessed on all weekend is going to shred your team's unoptimized vanilla Ruby or Java install you didn't even have time to write tests for and don't honestly care about because you're too worried about being laid off.
For use cases with simple business logic and high throughput, yeah the optimized Rust, maybe on your own hardware, makes sense.
Tons of cloud customers running NodeJS or Java stuff don't fit this. Maybe they have a few optimized pieces with high throughput and majority unoptimized where resource usage is low either way. Or they might be ok spending 10X compute resources even on something high-throughput because landing new features/products matters way more than optimizing a small part of their budget. Like, there are businesses renting big 8-socket servers with TBs of RAM to support some monstrous corporate DB that can only run on a single machine.
Yeah its kind of sad that we still have to use node and java with better options out there. Don't get me wrong, Typescript, Deno, JVM, etc.. are all doing their best to right those ships, but it's would still be better to just scrap that stuff for a better foundation.
That is what everyone knows,
so that is what's taught,
so that is what everyone knows,
so that is what's taught.
I'd gladly onboard projects to Erlang, Rust, Zig, Go, or anything modern. Would save our planet so much wasted energy.
The problem is that our industry democratized to the point that people want to hire the cheapest resources who can ship something. A mediocre node or ruby developer costs peanuts to hire compared to one who is mediocre at Go or Erlang. Btw.. If you hire a mediocre go or Erlang developer you’re fucked.
For IO bound work there’s not much difference between a service built with Go or Node. It’s really when you need services that have soft/hard real-time requirements or continuous processing requirements that imo it’s necessary to hire the right people for the right stuff.
For anyone else who is making a web platform that doesn’t have much complex backend processing, they can just use ruby or node.
Personally I use Go for all backend tasks and node for any one off scripting work. Go spits out a binary I can run on any Linux based system. Node scripts are easy to write and I use them for fixing issues or for simple single responsibility tasks.
Just gotta wait for the comment claiming admin/maintenance costs makes non-cloud more expensive than cloud then we can have yet another full repeat discussion.
Most services are probably well served by AWS/Heroku and others and simply saving the engineering time on developing the product, or optimizing when it becomes a problem
AWS can be 2x as expensive as DO for a simple VM, which you can easily work around by acquiring Reserve Instances or using "spot" instances
but, the "AWS premium" becomes a problem:
- 10x when dealing with bandwidth which you cannot do much
except a very big enterprise package and getting some discount that even then it will be multiple times higher than the alternatives
The way I see it, AWS is great but you need to consider your 1) key infra requirements
2) consider cost effectiveness of different solutions/services
3) consider alternatives i.e.
- Cloudfront is fine, but if you doing 100 tb/month probably Cloudflare would be best!
- Lambda is fine but if you doing 100M requests/month EC2/ECS would probably work better (or more cost effective)
Consider
1) what you need to get started and ship
2) what you need to scale up
3) what's the most cost effective solution long term
The requirements will change over time, be ready to adapt!
It looks clear to me that given the hardware they need, and the scale they operate at, buying their own metal makes sense. Also them sticking to similar servers is a good idea imho.
What I am missing though in the monthly cost is the personnel cost. I would assume that a decently qualified system and network administrator in Singapore, capable of operating and servicing this setup, will cost upward of $10,000 per month. And only having one is probably not enough, due to SPOF and other considerations. To be honest, even for the cloud you probably need someone to administer the setup, but chances are high that there is much less to do.
Paying for, and finding qualified staff is from my experience the main driver for small and mid-size companies to go to the cloud. And for many this actually makes sense financially. Even though many create unnecessary complex setups or get trapped in vendor lock-ins.
A small-medium DC can be a lot of work to install, specifically in terms of backup power/cooling, but not much to maintain on a daily basis.
Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required.
Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.
Building a DC is yet another level. Co-location facilities drop redundant power in your cabinet and manage cooling. All you need to do is make sure you plug your redundant power supplies into the right PDU. Also, any decent co-lo provider will rack and provide remote access to HW.
For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute.
For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate).
I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them.
That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.
Remote hands, maintenance contracts, and employees who will do network/firmware/firewall config, patching and maintenance aren’t included.
Also there’s no software costs… if you’re going to compare with the cloud you need to include VMware/Microsoft suite licenses, maintenance, and operators to have even remotely comparable capabilities. That tips this comparison strongly in favor of cloud.
Remote hands from co-lo facilities are typically free for simple activities like hitting a switch, plugging something in, etc. Hourly rates are extremely reasonable and initial racking is often included. Reality is if you’re using remote hands after the initial install you’re doing it wrong - out of band management, PXE boot to full machine bootstrap via ansible, K8, etc isn’t any more complicated than managing a bunch of VPCs, cloud firewalls, routing, cloud templates/terraform, etc.
Using Dell (as one example) you’d really be cheaping out on your hardware warranty if it didn’t include next day hardware replacement (for three/five years) and as I noted before, when placing server orders in the past they’ve discounted warranty/support to practically nothing. It’s actually fairly difficult to get a warranty that doesn’t include it. This is a level of support where the vendor literally dispatches someone to the facility who gets you back up and running. With redundant power supplies, RAID, etc the individual machine often doesn’t even go down and they just show up and replace the redundant component with the blinking light.
AWS is 20 years old and in two decades hardware has gotten EXTREMELY reliable and performant - in managing hundreds of servers across multiple facilities over the decades I can count the number of hardware failures on my fingers. The FUD/scare tactics/marketing of clouds to scare people away from hardware have been extremely effective. As one example for the most "unreliable" component the BackBlaze hard drive stats tell the story - they have a roughly 1% failure rate across > 100k consumer level mechanical drives[0] in service for a long as 92 months. These "failure prone" consumer drives aren't even available from server vendors. This is why their pricing is at least 1/10th that of big cloud.
VMware? Microsoft suite? For these kinds of applications I don’t know of anyone using anything other than containers, K8s, OpenStack, etc and Microsoft licensing is never “free” regardless of the approach. More traditional “enterprise applications” with a bunch of VMs that would necessitate something like VMware are essentially outside of scope and a different animal completely.
I also don’t know of any organizations outside of the smallest startups that don’t have at least one full time cloud devops/architect to manage anything beyond the simplest of cloud deployments. Compared to someone using an orchestration platform and managing a K8s control plane the employee cost is often a wash. For the rest of the dev/devops teams shipping a container is shipping a container.
The pricing differences at this scale are unbelievably dramatic (as noted in the article) - with ROI on hardware purchases (leases) and hosting vs cloud often less than one year. That’s A LOT of spend left over for the rounding error miscellaneous costs often noted. Cloud egress bandwidth pricing markups alone can pay for it when you’re pushing the kind of traffic that would warrant this approach.
I’ve been doing this since before AWS existed - the cloud has its place but in almost every case when this debate comes up arguments for “big cloud, always” typically come from people who have never deeply and holistically compared these approaches (and certainly never actually implemented “hardware”). For most of the "cloud only" generation their only experience with hardware is that time their laptop died and needless to say server grade hardware in temperature controlled facilities on conditioned power is infinitely more reliable than a consumer laptop getting thrown around.
Not saying that’s the case with you personally and “hardware, always” is just as inappropriate as “cloud, always” but to never even consider hardware and hosting in many cases results in wildly expensive cloud costs.
Don’t even get me started on cloud cost controls. The pricing is so dynamic and so obtuse there is an entire cotta...
> What I am missing though in the monthly cost is the personnel cost.
It's not like operating AWS doesn't involve any personnel cost, does it? Even though it was marketed as such, it has become clear that managing a public cloud such as AWS is at least as complex as managing a datacenter. Sure, you don't swap broken drives in RAID arrays but you have other complexities to deal with - and that complexity increases year by year.
I'm pretty confident that at small scale AWS requires less operating costs that colo. Things as simple as s3 take lots of work to setup and run in a colo (well probably at least 3 locations if you want really good durability) and things like backups quickly build on that. Even a single computer will have occasional hardware failures, system installs and more. You usually can't restore an entire disk from backup like you can on AWS if a DB upgrade goes wrong.
But I think AWS "management" definitely starts paying for itself when you want the higher availability features such as managed SQL databases, load balancers, queues and other features. None of them are super complicated on their own but they add up quickly, and at small scale the cost of having experts in each service will outweigh the cost of the service.
It is also worth considering the opportunity costs of changes. On-prem you may have a few spare servers lying around but if you need another machine or a new piece of hardware it will take significant time. On AWS you can spin up a 4 GPU machine in seconds, if you need more webserver machines just press the button. This quick turnaround can be very valuable as you are developing a new product and don't quite know what the requirements are.
As you get bigger the costs definitely flip. The profit margin that AWS takes starts to outweigh the costs of having more experts (and as your team grows you may have already acquired most of these experts, so no training is required) and it makes sense to move things in-house onto dedicated hardware.
Nope, senior devops / sys admin folk here in Singapore are more in the 7K - 8.5K (usd) range. Unless of course the employer is paying FFANG level salaries
> So we decided to consolidate all the costs spent for this data center, divide them by the number of servers, and compare them to the cost of a similar installation in the Amazon Web Services (AWS) cloud (we use this as the leading IaaS example).
so compare them to buying AWS VMs you don’t need? one of the main attractions of the cloud is the fact that you don’t have to pay for what you don’t use. Elasticity.
Every time I see one of these is: I bought 100 servers (which i use 40 of), I would’ve saved money if I was renting 100 servers of AWS. Yeah … no shit. If you were renting only 40, and had the ability to go down as usage decreases, there would be savings, mind you there are such things as, RIs, savings plans, etc, etc. meh.
Once your operation is stable and you know (almost) exactly what you need then sticking to the Cloud makes less and less sense. The premium they charge is enormous.
Not sure why e.g. Netflix doesn't exit AWS considering their needs flatlined. Or maybe they have some special agreement with much lower prices than market.
What the cloud gives you is the removal of the inter-department conflict between IT and Product Engineering. This is especially bad since IT is a cost-center rather than a revenue generator. Having a single provider who in turn has a single client is a horrible dynamic. It can work for the short term but long term you get really odd dynamics. Sure the cloud is expensive but so is the aggregate opportunity cost of waiting 3 months and spending 20 hours in meetings for a single dev server.
Hardware leasing/equipment financing is OpEx. The difference vs cloud is you’ll get much more capability at a fraction of the cost and own an asset at the end.
I talk about it a lot on HN but if you really want to get the financial types excited about leasing there are all kinds of interesting options like Section 179 Qualified Financing that can often sweeten the pot further.
>” Or maybe they have some special agreement with much lower prices than market.”
These exist and falls under the “Enterprise Discount Program”, but there are other PPAs (Private Pricing Addendum).
The idea with EDP is if you have qualifying on-demand use that not covered by a Savings Plan or Reserved Instance, you instead get the EDP discount applied to the line item.
Where I work is actively renegotiating its EDP discount. Right now we get 14% off for applicable on-demand usage.
There are also other programs. Such as the “Amazon Partner Network” “Workload Migration Program” (APN WMP) for SAAS companies, which are credits (lump sum, not per line item) that are awarded for moving customers to your SAAS products that are hosted on AWS infrastructure. These are big 6-digit chunks for us. Many other credits and programs as well.
There a fantasy that says you don’t need experts to run your cloud infrastructure.
The same fantasy says that if you own your own hardware you’ll need lots of experts that are so expensive and so hard to find that it makes no sense to self host.
Cloud is crazy expensive and delivers much slower computing power per dollar.
Consider if you need kubernetes plus cloud scaling, or you just need one modern ripping fast 16 core machine with 128GB RAM.
It's bottom lines like the article's where I wonder if all the companies who had HORRID penny-wise pound-foolish (6 months of paperwork to get a server, and only then would they order the servers) and went cloud simply did what these guys appear to be doing:
buy enough to stay ahead of demand. If you have unused servers, who cares, at least you don't have allocated by unused AWS servers that pay for themselves in hardware costs every... four months? I don't remember how long it is.
I sold all my AMZN stock by 2006 when I quit thinking that everyone would figure out the secret to doing your own hardware cheap.
I then went to try to convert other businesses over to doing it cheaply, but as an engineer I just wound up getting into fights with my managers who loved spending money.
It is all about Empire building and managers wanted those expensive VMWare and EMC vendor relationships to manage in order to fluff up their resumes. They want more people under their umbrella.
If you start hiring Directors of Network Engineering you'll find that you start hiring lots of expensive CCIEs and spending a lot of money on Cisco.
It is almost a problem of being too small, since you can't quite easily point out how having single SATA drives in all the servers will save millions over having hardware RAID cards, and having simple networking in your rack design will also save millions over what Cisco would like to push on you. So you overspend by $250k here and there and everywhere, and pretty quickly you're back up to cloud costs.
At a smaller level it isn't quite as obvious how obscenely expensive to your capex these kinds of managers are, and they know hot to talk in ways that make Managment believe that they're buying all kinds of Enterprise-class resiliancy and leveragable vendor relationships and all that kind of nonsense. Hiring smart engineers instead is something that most Managers don't know how to do either, which is what you need in order to get a decent level of automation in place in order to scale the ability to manage it all.
And don't get me started on high priced consultants that can be brought in to overdesign the most complicated solutions possible by never saying 'no' to any kind of requirement that IT management can dream up. I've seen VLAN diagrams that you wouldn't believe...
But anyway, if you can solve that human problem, through hiring the right people and having a corporate culture of "Frugality" (as much as I hate it, that focus Amazon has is what let them do IT cheaply and then resell it). But it isn't a technical problem, or a financial problem, or even really a staffing problem, it is fundamentally a culture problem.
And then the trick in my mind would be to do Frugality right without being as much of a dick about it as Amazon is.
45 comments
[ 4.9 ms ] story [ 103 ms ] threadIf you have an 8GB vm you can see it in action via https://docs.openstack.org/devstack/zed/guides/single-vm.htm... (takes about 45 minutes) or if you want the "full" experience fire up an education cluster from https://openmetal.io/use-cases/on-demand-openstack-cloud/ (it takes about 4 hours or so but it's also a "real" setup unlike the vm which for sure allows running instances via qemu but isn't going to allow exercising the complete suite as easily)
Think of how many copies of the data you need (and how much processing it takes) to use:
- a L3 CPU cache.
- a 16GB RAM cache.
- an embedded C/Rust/Go SQLite database.
- a distributed, fault tolerant Cassandra/elastic cache cluster.
You can literally solve the same typeahead/autocomplete/user lookup/session management/etc.. features using the last three of these - but the CPU cycle, memory usage and latency costs (plus complexity) explodes.
I'm always impressed by people that handle 1 million connections on a single box while I'm watching teams trying to handle a thousand connections with a whole fleet of infrastructure and caching layers.
Then again, part of the issue is the wrong language and wrong designs patterns. Highly optimized Rust and Go someone obsessed on all weekend is going to shred your team's unoptimized vanilla Ruby or Java install you didn't even have time to write tests for and don't honestly care about because you're too worried about being laid off.
Tons of cloud customers running NodeJS or Java stuff don't fit this. Maybe they have a few optimized pieces with high throughput and majority unoptimized where resource usage is low either way. Or they might be ok spending 10X compute resources even on something high-throughput because landing new features/products matters way more than optimizing a small part of their budget. Like, there are businesses renting big 8-socket servers with TBs of RAM to support some monstrous corporate DB that can only run on a single machine.
That is what everyone knows,
so that is what's taught,
so that is what everyone knows,
so that is what's taught.
I'd gladly onboard projects to Erlang, Rust, Zig, Go, or anything modern. Would save our planet so much wasted energy.
For IO bound work there’s not much difference between a service built with Go or Node. It’s really when you need services that have soft/hard real-time requirements or continuous processing requirements that imo it’s necessary to hire the right people for the right stuff.
For anyone else who is making a web platform that doesn’t have much complex backend processing, they can just use ruby or node.
Personally I use Go for all backend tasks and node for any one off scripting work. Go spits out a binary I can run on any Linux based system. Node scripts are easy to write and I use them for fixing issues or for simple single responsibility tasks.
This doesn't sound like a problem to me.
https://news.ycombinator.com/item?id=35108429
https://news.ycombinator.com/item?id=35100866
https://news.ycombinator.com/item?id=35095604
https://news.ycombinator.com/item?id=35094407
Most services are probably well served by AWS/Heroku and others and simply saving the engineering time on developing the product, or optimizing when it becomes a problem
AWS can be 2x as expensive as DO for a simple VM, which you can easily work around by acquiring Reserve Instances or using "spot" instances
but, the "AWS premium" becomes a problem: - 10x when dealing with bandwidth which you cannot do much except a very big enterprise package and getting some discount that even then it will be multiple times higher than the alternatives
The way I see it, AWS is great but you need to consider your 1) key infra requirements 2) consider cost effectiveness of different solutions/services 3) consider alternatives i.e.
- Cloudfront is fine, but if you doing 100 tb/month probably Cloudflare would be best! - Lambda is fine but if you doing 100M requests/month EC2/ECS would probably work better (or more cost effective)
Consider 1) what you need to get started and ship 2) what you need to scale up 3) what's the most cost effective solution long term
The requirements will change over time, be ready to adapt!
I am free to switch vendor every 12 months without pain.
I like Cloud66 because:
- Pricing is fair. I spend 100$/month for my stack - They back up db and host it to a different cloud provider (AWS)
- It’s easy to import Db
- UI is clean and easy to use
- It’s easy to switch vendor. Just clone the app and it will install all in a new server.
I have switched from Digital Ocean to Azure in a couple of hours with 30 min downtime.
What I am missing though in the monthly cost is the personnel cost. I would assume that a decently qualified system and network administrator in Singapore, capable of operating and servicing this setup, will cost upward of $10,000 per month. And only having one is probably not enough, due to SPOF and other considerations. To be honest, even for the cloud you probably need someone to administer the setup, but chances are high that there is much less to do.
Paying for, and finding qualified staff is from my experience the main driver for small and mid-size companies to go to the cloud. And for many this actually makes sense financially. Even though many create unnecessary complex setups or get trapped in vendor lock-ins.
Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required.
Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.
For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute.
For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate).
I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them.
That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.
Also there’s no software costs… if you’re going to compare with the cloud you need to include VMware/Microsoft suite licenses, maintenance, and operators to have even remotely comparable capabilities. That tips this comparison strongly in favor of cloud.
Using Dell (as one example) you’d really be cheaping out on your hardware warranty if it didn’t include next day hardware replacement (for three/five years) and as I noted before, when placing server orders in the past they’ve discounted warranty/support to practically nothing. It’s actually fairly difficult to get a warranty that doesn’t include it. This is a level of support where the vendor literally dispatches someone to the facility who gets you back up and running. With redundant power supplies, RAID, etc the individual machine often doesn’t even go down and they just show up and replace the redundant component with the blinking light.
AWS is 20 years old and in two decades hardware has gotten EXTREMELY reliable and performant - in managing hundreds of servers across multiple facilities over the decades I can count the number of hardware failures on my fingers. The FUD/scare tactics/marketing of clouds to scare people away from hardware have been extremely effective. As one example for the most "unreliable" component the BackBlaze hard drive stats tell the story - they have a roughly 1% failure rate across > 100k consumer level mechanical drives[0] in service for a long as 92 months. These "failure prone" consumer drives aren't even available from server vendors. This is why their pricing is at least 1/10th that of big cloud.
VMware? Microsoft suite? For these kinds of applications I don’t know of anyone using anything other than containers, K8s, OpenStack, etc and Microsoft licensing is never “free” regardless of the approach. More traditional “enterprise applications” with a bunch of VMs that would necessitate something like VMware are essentially outside of scope and a different animal completely.
I also don’t know of any organizations outside of the smallest startups that don’t have at least one full time cloud devops/architect to manage anything beyond the simplest of cloud deployments. Compared to someone using an orchestration platform and managing a K8s control plane the employee cost is often a wash. For the rest of the dev/devops teams shipping a container is shipping a container.
The pricing differences at this scale are unbelievably dramatic (as noted in the article) - with ROI on hardware purchases (leases) and hosting vs cloud often less than one year. That’s A LOT of spend left over for the rounding error miscellaneous costs often noted. Cloud egress bandwidth pricing markups alone can pay for it when you’re pushing the kind of traffic that would warrant this approach.
I’ve been doing this since before AWS existed - the cloud has its place but in almost every case when this debate comes up arguments for “big cloud, always” typically come from people who have never deeply and holistically compared these approaches (and certainly never actually implemented “hardware”). For most of the "cloud only" generation their only experience with hardware is that time their laptop died and needless to say server grade hardware in temperature controlled facilities on conditioned power is infinitely more reliable than a consumer laptop getting thrown around.
Not saying that’s the case with you personally and “hardware, always” is just as inappropriate as “cloud, always” but to never even consider hardware and hosting in many cases results in wildly expensive cloud costs.
Don’t even get me started on cloud cost controls. The pricing is so dynamic and so obtuse there is an entire cotta...
It's not like operating AWS doesn't involve any personnel cost, does it? Even though it was marketed as such, it has become clear that managing a public cloud such as AWS is at least as complex as managing a datacenter. Sure, you don't swap broken drives in RAID arrays but you have other complexities to deal with - and that complexity increases year by year.
But I think AWS "management" definitely starts paying for itself when you want the higher availability features such as managed SQL databases, load balancers, queues and other features. None of them are super complicated on their own but they add up quickly, and at small scale the cost of having experts in each service will outweigh the cost of the service.
It is also worth considering the opportunity costs of changes. On-prem you may have a few spare servers lying around but if you need another machine or a new piece of hardware it will take significant time. On AWS you can spin up a 4 GPU machine in seconds, if you need more webserver machines just press the button. This quick turnaround can be very valuable as you are developing a new product and don't quite know what the requirements are.
As you get bigger the costs definitely flip. The profit margin that AWS takes starts to outweigh the costs of having more experts (and as your team grows you may have already acquired most of these experts, so no training is required) and it makes sense to move things in-house onto dedicated hardware.
so compare them to buying AWS VMs you don’t need? one of the main attractions of the cloud is the fact that you don’t have to pay for what you don’t use. Elasticity.
Every time I see one of these is: I bought 100 servers (which i use 40 of), I would’ve saved money if I was renting 100 servers of AWS. Yeah … no shit. If you were renting only 40, and had the ability to go down as usage decreases, there would be savings, mind you there are such things as, RIs, savings plans, etc, etc. meh.
Not sure why e.g. Netflix doesn't exit AWS considering their needs flatlined. Or maybe they have some special agreement with much lower prices than market.
I talk about it a lot on HN but if you really want to get the financial types excited about leasing there are all kinds of interesting options like Section 179 Qualified Financing that can often sweeten the pot further.
I think many companies simply get away by having unhealthy oncall rotations. E.g. ask people to oncall outside business hours constantly.
These exist and falls under the “Enterprise Discount Program”, but there are other PPAs (Private Pricing Addendum).
The idea with EDP is if you have qualifying on-demand use that not covered by a Savings Plan or Reserved Instance, you instead get the EDP discount applied to the line item.
Where I work is actively renegotiating its EDP discount. Right now we get 14% off for applicable on-demand usage.
There are also other programs. Such as the “Amazon Partner Network” “Workload Migration Program” (APN WMP) for SAAS companies, which are credits (lump sum, not per line item) that are awarded for moving customers to your SAAS products that are hosted on AWS infrastructure. These are big 6-digit chunks for us. Many other credits and programs as well.
The same fantasy says that if you own your own hardware you’ll need lots of experts that are so expensive and so hard to find that it makes no sense to self host.
Cloud is crazy expensive and delivers much slower computing power per dollar.
Consider if you need kubernetes plus cloud scaling, or you just need one modern ripping fast 16 core machine with 128GB RAM.
buy enough to stay ahead of demand. If you have unused servers, who cares, at least you don't have allocated by unused AWS servers that pay for themselves in hardware costs every... four months? I don't remember how long it is.
I then went to try to convert other businesses over to doing it cheaply, but as an engineer I just wound up getting into fights with my managers who loved spending money.
It is all about Empire building and managers wanted those expensive VMWare and EMC vendor relationships to manage in order to fluff up their resumes. They want more people under their umbrella.
If you start hiring Directors of Network Engineering you'll find that you start hiring lots of expensive CCIEs and spending a lot of money on Cisco.
It is almost a problem of being too small, since you can't quite easily point out how having single SATA drives in all the servers will save millions over having hardware RAID cards, and having simple networking in your rack design will also save millions over what Cisco would like to push on you. So you overspend by $250k here and there and everywhere, and pretty quickly you're back up to cloud costs.
At a smaller level it isn't quite as obvious how obscenely expensive to your capex these kinds of managers are, and they know hot to talk in ways that make Managment believe that they're buying all kinds of Enterprise-class resiliancy and leveragable vendor relationships and all that kind of nonsense. Hiring smart engineers instead is something that most Managers don't know how to do either, which is what you need in order to get a decent level of automation in place in order to scale the ability to manage it all.
And don't get me started on high priced consultants that can be brought in to overdesign the most complicated solutions possible by never saying 'no' to any kind of requirement that IT management can dream up. I've seen VLAN diagrams that you wouldn't believe...
But anyway, if you can solve that human problem, through hiring the right people and having a corporate culture of "Frugality" (as much as I hate it, that focus Amazon has is what let them do IT cheaply and then resell it). But it isn't a technical problem, or a financial problem, or even really a staffing problem, it is fundamentally a culture problem.
And then the trick in my mind would be to do Frugality right without being as much of a dick about it as Amazon is.