I know people are itching to jump to the conclusion that bare metal is ultimately way cheaper than cloud, but really, it isn’t.
At the end of the day, we want operating expenses, not capital expenses.
And if you purchase a ton of bare metal and suddenly your business needs change it will not be easy or quick to move that much metal or install new metal in place.
Why not both? Most businesses have a fairly stable baseload need for computing. Replacing EC2 instances with bare metal to satisfy that seems to make sense to me.
Eh. Most businesses don’t really need big scale. I build exclusively in the cloud, but the app I maintain could easily run on a single $2k server, with the possible exception of image resizing.
2 years ago we build thumbor-like apps in python using vibora and opencv. A single aws m8 large($80/month?) can serve hundred millions images resizing in a month.
At that time we build it with only 2 people, copy pasting code from internet in a few days and vibora’s github readme also state that its in alpha release.
With $2K server, you need to cheat to make those inefficient implementation :)
Side note:
Several weeks later I found out about thumbor and replace our ‘emergency work’ with dockerized thumbor
I also tinkered with thumbot in aws lambda. But the cost raised to $500/month. I think we did something wrong with aws rekognition.
The cost per query difference of cloud versus bare metal, in the article is over ten times. That kind of margin gives a lot of room for error. You could literally over provision by an order of magnitude and come out on top.
There's plenty of hidden costs though of doing this yourself that need a certain level of scale to pay off. Like people to manage the hardware, that's a big one.
Still renting bare metal servers from e.g. Packet could make much more sense even for small companies than using AWS.
renting small servers...much more sense even for small companies
No. At one time over 10 years ago, I had a dedicated server with one of the hosting companies. They charged me $225/month.
I moved to AWS and my cost dropped to ~$30/month. Even now, 10 years later, my total AWS bill for primarily EC2 and RDS services that runs an application that pays my bills is ~$75 per month.
If my company suddenly went viral and had a bunch of new customers, I could scale up instantly.
I'm not talking about hobby projects, which also aren't a great fit for AWS. Use digital ocean, vultr, or linode. I'm talking about startups with over $10k/year in spend.
> we want operating expenses, not capital expenses
Ultimately it's all capex. The choice is whether the capital asset is on your balance sheet or someone else's.
If someone else is tying up their capital in a depreciating and inflexible asset, they'll demand to be paid well for giving you the flexibility to avoid all of that.
So you can't really say opex is always better than capex, since it depends on the price you're willing to pay for that flexibility.
Ultimately whether or not a company should be using bare metal vs. the cloud is actually a financial decision, not a technical one. There are some technical decisions that can be made to reduce the risk of changes and/or a mixed use case, but ultimately the difference for most businesses is going to be up to the CFO.
The article makes a good point, that it is not only a financial decision (b/s vs pnl, risk premium etc) but as well an architectural and functional one. No CFO usually has a good enough understanding of the intricacies of those risks.
That’s fair, although wise architectural choices can help isolate the technical and financial decisions from each other. For example I think this is the strongest argument for k8s; it gives you the ability to isolate your compute from the underlying infrastructure.
You are creating a false dichotomy. If you aren't a mature company like expensify that can pretty accurately predict your needs, then you can rent bare metal for nearly as cheap as buying it. Probably cheaper if you don't have any special needs and commodity hardware will do.
It's not just the server cost that matters when you compare the cloud to a bare-metal server.
What about:
Networking equipment
Backup power - UPS and generators
Internet backbone - Speed as well as reliability
Cooling and other infrastructure costs
Then there is capital lock in risk. Once you buy the above hardware, you are probably stuck with it for a few years.
If you have stable load now for 25 big bare-metal servers, it may be cheaper to go bare metal. Otherwise, there is a lot more involved in the decision than going to dell.com and pricing a server.
Bare metal is a continuum too. You don't have to pick between a full cloud deploy and building your own datacenter from scratch.
You can get bare metal servers from a host on monthly or hourly terms.
You can get a 1/3rd rack cabinet at a colo facility and bring your own router, and use the colo's IPs
You can get a cage or a room at a colo and arrange for connecticity to your various upstreams and peers with your own ASN and IPs.
Only after all this is exhausted would I move towards and owned and operated datacenter where all the pieces are under company control and responsibility. (Although, if you rely on it, you're responsible for it, regardless of if you actually control it)
>As far as reliability, you can't compare almost any COLO location with an AWS data center.
Why not? We had much less downtime than AWS' NoVa DC while coloing, and I don't think our DC was exceptional. AWS is probably much more competent, but they're also trying to build something much more complex than your average small-scale colo setup.
If you're running bare metal on someone else's servers, you basically have the same staffing requirements as running on EC2. Just need people who understand real machines instead of virtual machines. You'll be paying the host instead of amazon for services and equipment. But the prices might be less.
There's no super secret reliability technology. You can absolutely compare colo with AWS. Many colos may be less reliable, and some may be more reliable. You need to look at power redundancy: how many utility feeds, what kind of ups, generators, testing schedules, what the points of failur are, how they responded in the past, etc. Same for networking feeds. Most colos have simpler networks than AWS which reduces the possible failures, but they may be less redundant etc; they probably have better bandwidth prices though.
Anyway, if you want a reliable system, you're going to need to be in multiple locations, and if you're in multiple locations, you should be able to weather the inevitable power relay failures and border router failures, etc every couple of years.
There are plenty of hosting providers that will provide everything you mention plus the actual servers for a monthly fee without requiring you to buy anything.
I agree but now the cost climbs. I am not saying AWS is cheap. Just that you need to look at all the services and components that support a server in a rack.
All included, still not even close. Not by a long shot. Renting bare metal from a major provider is going to be less than 20% the cost of aws and that's not even including the bandwidth which is even cheaper from bare metal vs aws.
There is no calculus that makes aws cheaper unless you need extreme variability or some of their managed services and even then, you can usually use those services in conjunction with your rented bare metals.
Trying to be helpful and not pedantic: "On premises" or "on prem" are good choices here, while "premise" means something else (e.g. "The premise of moving off-premises is that we can pay a little more in order not to worry about stuff that isn't our company's primary business.")
This is not the first time I have seen this but it is actually quite commonly used / mistake. I wonder if 20 years from now the word premise would have a different meaning.
Network alone is enough to pay for on-prem. Amazon egress fees are beyond egregious.
The only reasons I've seen to justify putting something in the cloud are:
You're tiny - colocation is fine but generally speaking if you don't need more than a single-server's worth of performance and you've got a for-profit business, "the cloud" is probably easier.
Your workload is extremely bursty.
You're using the other stuff. AWS/Azure/GCP has all sort of services that require scale to be efficient - if you're subscribing to those services and are medium-sized (I know, that's not even loosely defined) - it probably makes sense to be in AWS/Azure/GCP.
If you don't need those services, or you're extremely large, you can almost always do it cheaper yourself. Whether you want to is another/completely different question.
You can rent a EPYC Server 32c/64t with 2x4TB NVMe and 512GB memory and 10Gbit (20TB free traffic - 1,20€/TB above that) port in a Hetzner datacentre (cooling, backbone, backup power, network all managed professionally) for ~450€/month.
The time from ordering to login is something between 15 minutes and 2-3 days (you know beforehand). You can cancel every month. Faulty disks are replaced 24/7 in 15-30 minutes.
In the list of features which are somewhat custom to their use case, they listed "Disable POSIX advisory locks".
Turns out this is available stock right now as the "unix-none" VFS implementation. We're using it in an almost-but-not-quite-POSIX RTOS that happens to be missing advisory locks.
The randomization they are working around is most likely
> WHERE indexedColumn > RANDOM() LIMIT 10
That can be bound on RANDOM() performance or we can rely on a seeded "non-random" PRNG to produce a random selection vector without running random() for every row.
If that has a syscall into /dev/urandom for every row, then that sucks for sure & is not representative of the actual intention.
In Apache Hive, we have a similar argument about unix_timestamp(), which was made deterministic, but stateful (MapReduce failure tolerance demands that if a task fails while running a query like > unix_timestamp() + n, the next attempt will produce an identical output, even-though some time has passed between the attempts).
So for that, we do the unix_timestamp() replacement per-query, instead of per-row.
A deterministic random() is good for reproducing tests. What you want is a way to use it to generate a pseudorandom output based on "more random" input, such as /dev/urandom.
They're comparing bare metal to a virtual machine here, aren't they? Of course I'd expect some pretty noticeable performance differences there. Not sure they were offered at the time of the post but AWS also has bare metal that seems like it'd but a much more fair comparison
Cloud VM vs on-prem bare metal doesn't seem like a very fair comparison
43 comments
[ 0.22 ms ] story [ 109 ms ] threadAlso, it's already been submitted a bunch of times:
https://news.ycombinator.com/item?id=23070888
https://news.ycombinator.com/item?id=22856746
https://news.ycombinator.com/item?id=20778477
https://news.ycombinator.com/item?id=16118776
At the end of the day, we want operating expenses, not capital expenses.
And if you purchase a ton of bare metal and suddenly your business needs change it will not be easy or quick to move that much metal or install new metal in place.
Stick to the cloud.
2 years ago we build thumbor-like apps in python using vibora and opencv. A single aws m8 large($80/month?) can serve hundred millions images resizing in a month.
At that time we build it with only 2 people, copy pasting code from internet in a few days and vibora’s github readme also state that its in alpha release. With $2K server, you need to cheat to make those inefficient implementation :)
Side note: Several weeks later I found out about thumbor and replace our ‘emergency work’ with dockerized thumbor
I also tinkered with thumbot in aws lambda. But the cost raised to $500/month. I think we did something wrong with aws rekognition.
There's plenty of hidden costs though of doing this yourself that need a certain level of scale to pay off. Like people to manage the hardware, that's a big one.
Still renting bare metal servers from e.g. Packet could make much more sense even for small companies than using AWS.
No. At one time over 10 years ago, I had a dedicated server with one of the hosting companies. They charged me $225/month.
I moved to AWS and my cost dropped to ~$30/month. Even now, 10 years later, my total AWS bill for primarily EC2 and RDS services that runs an application that pays my bills is ~$75 per month.
If my company suddenly went viral and had a bunch of new customers, I could scale up instantly.
Ultimately it's all capex. The choice is whether the capital asset is on your balance sheet or someone else's.
If someone else is tying up their capital in a depreciating and inflexible asset, they'll demand to be paid well for giving you the flexibility to avoid all of that.
So you can't really say opex is always better than capex, since it depends on the price you're willing to pay for that flexibility.
What about:
Then there is capital lock in risk. Once you buy the above hardware, you are probably stuck with it for a few years.If you have stable load now for 25 big bare-metal servers, it may be cheaper to go bare metal. Otherwise, there is a lot more involved in the decision than going to dell.com and pricing a server.
You can get bare metal servers from a host on monthly or hourly terms.
You can get a 1/3rd rack cabinet at a colo facility and bring your own router, and use the colo's IPs
You can get a cage or a room at a colo and arrange for connecticity to your various upstreams and peers with your own ASN and IPs.
Only after all this is exhausted would I move towards and owned and operated datacenter where all the pieces are under company control and responsibility. (Although, if you rely on it, you're responsible for it, regardless of if you actually control it)
As far as reliability, you can't compare almost any COLO location with an AWS data center.
Why not? We had much less downtime than AWS' NoVa DC while coloing, and I don't think our DC was exceptional. AWS is probably much more competent, but they're also trying to build something much more complex than your average small-scale colo setup.
There's no super secret reliability technology. You can absolutely compare colo with AWS. Many colos may be less reliable, and some may be more reliable. You need to look at power redundancy: how many utility feeds, what kind of ups, generators, testing schedules, what the points of failur are, how they responded in the past, etc. Same for networking feeds. Most colos have simpler networks than AWS which reduces the possible failures, but they may be less redundant etc; they probably have better bandwidth prices though.
Anyway, if you want a reliable system, you're going to need to be in multiple locations, and if you're in multiple locations, you should be able to weather the inevitable power relay failures and border router failures, etc every couple of years.
There are plenty of hosting providers that will provide everything you mention plus the actual servers for a monthly fee without requiring you to buy anything.
There is no calculus that makes aws cheaper unless you need extreme variability or some of their managed services and even then, you can usually use those services in conjunction with your rented bare metals.
Trying to be helpful and not pedantic: "On premises" or "on prem" are good choices here, while "premise" means something else (e.g. "The premise of moving off-premises is that we can pay a little more in order not to worry about stuff that isn't our company's primary business.")
Very common, for sure. And although I think it's worth professionals being aware of the difference, I tend not to bother with less technical folks.
> I wonder if 20 years from now the word premise would have a different meaning.
Given that "literally" is now often used to mean "figuratively", it's absolutely possible!
The only reasons I've seen to justify putting something in the cloud are:
You're tiny - colocation is fine but generally speaking if you don't need more than a single-server's worth of performance and you've got a for-profit business, "the cloud" is probably easier.
Your workload is extremely bursty.
You're using the other stuff. AWS/Azure/GCP has all sort of services that require scale to be efficient - if you're subscribing to those services and are medium-sized (I know, that's not even loosely defined) - it probably makes sense to be in AWS/Azure/GCP.
If you don't need those services, or you're extremely large, you can almost always do it cheaper yourself. Whether you want to is another/completely different question.
The time from ordering to login is something between 15 minutes and 2-3 days (you know beforehand). You can cancel every month. Faulty disks are replaced 24/7 in 15-30 minutes.
Turns out this is available stock right now as the "unix-none" VFS implementation. We're using it in an almost-but-not-quite-POSIX RTOS that happens to be missing advisory locks.
See also https://sqlite.org/vfs.html
> WHERE indexedColumn > RANDOM() LIMIT 10
That can be bound on RANDOM() performance or we can rely on a seeded "non-random" PRNG to produce a random selection vector without running random() for every row.
If that has a syscall into /dev/urandom for every row, then that sucks for sure & is not representative of the actual intention.
In Apache Hive, we have a similar argument about unix_timestamp(), which was made deterministic, but stateful (MapReduce failure tolerance demands that if a task fails while running a query like > unix_timestamp() + n, the next attempt will produce an identical output, even-though some time has passed between the attempts).
So for that, we do the unix_timestamp() replacement per-query, instead of per-row.
I assume that's what was done here.
Cloud VM vs on-prem bare metal doesn't seem like a very fair comparison