It's very short sighted to ignore the money being saved by not having to worry about all those things. Their "$5k" example is probably wrong and also irrelevant in most (enterprise) contexts.
Or perhaps just an experienced sysadmin who has different opinions and preferences than you do. There is never a single TheRightWayToDoThings(tm), different people have different needs, priorities and/or resources available.
Hard to take seriously and barely rises to the level necessary for rational debate. Is there a workload that is demonstrably 100x slower on AWS or GCE compared to the fully-loaded cost of bare metal?
MySQL-RDS is the AWS service which I felt gave me the least value for the dollar. (Except for the OpenVPN server which went nuts and did an obscene number of I/Os against EBS which ran up my bill)
"Well optimized MySQL" sounds a bit like having a really attractive liver cancer. Cloud services expand your options and it's worth pondering them. Don't ask what is the best MySQL, ask for a given amount of items I need to store and recall what is the best server OR SERVICE that could do it? MySQL on bare metal? On virtual machines? RDS? Cloud Spanner?
Do cloud providers increase your ability to choose scale-up instead of scale-out? Instead of your sharded and replicated MySQL that is almost inoperable and never seems to maintain consistency, would you be better served with one gigantic database? Sure it costs $13k/month to rent one from Google but it costs a quarter million to buy one from Dell. These are all aspects of the decision-making process.
The author was largely focused on specific implementations over architecture patterns, so I wouldn’t think they’re operating at a level where TCO is a consideration...
The TechEmpower benchmarks showed roughly a 10x difference in performance between AWS and bare-metal for basic webserving + DB, across many different languages and frameworks:
OK but that's a comparison of a $3200 machine with 7x more CPU and more than double the memory of the $100/month virtual machine. The operating costs of the metal machine (electricity, repairs) and its lifetime are unstated.
to be fair techempower benchmarks are not really "world comparable".
it's still just a benchmark. I know a lot of people take these benchmarks as their bread and butter, but there are so many "perf" hacks in them and they do not really have a lot in common with real world usage.
Didn't you see? You're just supposed to learn some sysadmin skills yourself.
The trick is to make yourself the linchpin, so that once the site grows large enough you can never leave, as you're the only one able to keep that writhing mass of wires up and running.
Yeah, been there, though against my will. Boy was I happy when the system (which actually worked extremely well) was shut down and replaced with a SaaS-based product.
Indeed. Even despite all the shiny web console controls, someone has to know what all the items on the invoice from the cloud provider do, and how they work together.
I actually have had physical boxes with Hetzner, as suggested by the author, for over 10 years, and ran multiple projects on them. Of course I had my share of disk failures. You tell them the disk serial via a special form, tell them when the necessary downtime for the swap is least disturbing for you, and someone switches the disk as requested. 5 minutes later the machine is up again, you trigger RAID rebuild, and that's it.
Doesn't cost a dime to do this, and not more than 20 minutes of work and 10 minutes downtime from my side.
The exact monetary cost of work time and application downtime (if there even is any, maybe you just shift the load to the second box) varies greatly depending on the context in which the respective project is run (hobby vs. paid-for, I've had both types), which is why I offered time estimates for these costs instead of a monetary amount. The stated monetary amount of "zero" refers to the provider not charging me anything for the new disk itself and the work of changing the disk.
There's common consent that the lowest bar for some hosting to qualify for the "cloud" moniker is that it must abstract from the physical hardware. Most people would even add that it is necessary for some kind of management system to exist which allows for _automated_ provision of your resources.
Renting bare metal instead of owning does not meet this lowest bar. Getting some remote hands service on top with your rental does not change this classification.
There more benefit than that. One example...development teams can't force the same level of tech debt that they can on-prem. If Amazon's upgrading some service or competent, there isn't any bargaining or exceptions.
Many similar benefits when you have a huge for-profit entity running the infrastructure at their scale. They optimize a lot for you.
But you usually also have to wait longer for new versions of services -- which might not be a problem for the majority, but can be a show-stopper for some.
AWS almost never forcibly spins down legacy functionality, even if they stop advertising it, unless literally no one is left using it.
That's actually a selling point of AWS to enterprise: while idealistic devs may like “our cloud vendor will force us to use the latest and greatest so we don't have to have that fight”, enterprise buyers don't want “we’re going to be compelled to reengineer our working systems on someone else's timetable”.
> it’s in not having to pay a sysadmin so you can work on business problems.
...
> that benefit only improves with scale.
That's hard to derive from the same statement - there's a point after which you can afford your own 8 member SRE team across 3 timezones.
The cloud is incredibly attractive when you consider your scale problems as unknown (as a startup, the blitz scale day isn't a good time to go rack nodes yourself) & mostly as way to scale down the costs faster than physical infrastructure + associated human costs can.
I've started looking at this as slightly different model - I'm no longer hiring a good operations manager for those 8 people, I'm hiring a decent programmer who can automate and have code handling deployment tasks instead of a human being taking orders (i.e rack an NVMe + reboot this).
Honestly, even with your physical infrastructure, the trend is towards API based deployment models.
My time at Zynga working on ZCloud, migrating off EC2 to on-prem, that is what I saw.
The API based deployment model + reactions from machines (on failure), was worth building everywhere - now that applies to k8s.
> The money you save in the cloud isn’t on hardware, it’s in not having to pay a sysadmin so you can work on business problems
Maybe, but you probably have to go pretty far on the cloud maturity spectrum (using cloud native services effectively, not simple VMs like EC2 or even comparatively thin services that abstract just the machine/OS layer like non-Aurora RDS) before you are saving more in local-machine-focused ops teams than you are paying in cloud-focussed ops teams. The latter are more likely to be timeslices of time of team members that are primarily developers, but its still work with a cost.
I dunno. Going from "lift-and-shift" use of EC2 to Cloud Native (lambda, cloudfront, etc.) lowered my AWS bill from $300 a month to $50 a month.
I switched from the dedicated hosting that he advocates to AWS long ago because I was always having things get broken by the people at Softlayer. For instance one time I upgraded my network port and somehow that broke my record in the issue tracking system so I could not longer put issues (or any work reports in.)
There was the clunky and expensive backup server (I'm not sure if it would really restore...) and the breaking point was when somebody added a new disk but they did it wrong so that the partition table got overwritten when the machine rebooted. I was able to fix the partition table but then I moved my files to EC2 as fast as I could because i didn[t have time to deal with that.
As for queues I wonder what is up with that. When I build systems based on message queues they work OK, but I've got an eye for detail and for architecture that many people don't seem to have because I always see other people get in trouble with them.
To be fair, those are cheaper because they want to make you more reliant on AWS and because those are more idyllic for Amazon to run, as with Lyft. Same with AWS Activate and etc.
But for most people, it's simpler to just use them and get it over with, and I'm glad it worked for you.
Yeah, I think the kicker with managed services vs. self-hosted is whether you're paying for capacity or by usage. The thing with capacity driven costs is that you will always have some unused capacity (and indeed you should as a safety measure). If you're paying by usage, then that pushes the issue of managing overhead capacity onto the service provider.
The core benefit of "the cloud" is missed here. I don't think any serious engineers are saying you CAN'T do it this way, just that nobody wants to anymore.
Respectfully disagree. I'm an ops person, and I have a nostalgic attachment to running my own stuff in boxes that I can touch (or at least visit). However, cloud is just better for most applications. Certainly there are some scenarios where rolling your own is important, but it is uncommon now and only going to increase.
Yes cloud is expensive in a raw performance per $ sense, but that isn't it's value proposition. Most workloads aren't limited by the processing power that you can buy anyway. The draw for cloud is that it abstracts away entire classes of problems. Certainly it provides new problems as well, but those can be worked on in concert with everyone else working on the same platform.
Man I wish HN had a downvote option for articles that are just not worth reading.
"Don't go to the cloud, just buy your own servers" completely ignores the reason anyone rents cloud servers in the first place. If I could easily say "I need exactly this much capacity, no more no less, with no unexpected scaling needs and no code/infrastructure changes" then I'd be sitting pretty. Now for a show of hands, how many companies does this describe?
And of those companies that raised their hand, how many can say "I'm fine with just having a server in Germany" who will then go on to say "and I don't need a CDN to serve customers in other regions"?
>keep running live/live setup...This is very critical
Oh I didn't know it was that easy. If all you have to do to keep a server running is say "stay running, it's very critical" then of course no one needs any managed services. Outages are solved forever.
Completely worthless advice from a text file written by... I'm sorry who is this person and what authority do they have?
not buying into aws and not having a cdn are completely different questions. I'd downvote your comment even before reaching the ad hominem attack at the end.
A $200 DIY hardware cost (equivalent cloud price quoted as $5000 in the article) suggests that, for $4800/mo, you too can do Dev Ops yourself. For a lot of people, that proposition is a hard pass. You want enterprise-grade backups. If your availability drops, your business will lose credibility.
Also, good luck finding a Dev Ops engineer for $57,600 a year.
Not even really 57,600$. Over here, a person costs their company 1.5x the salary written in their contract (and nearly 2x their "take-home" salary). And most developed countries have something similar (more or less).
So the salary we're really talking about actually is less than 40k a year.
Good luck finding a Cloud Engineer for that money to manage your cloud-based infrastructure. ;) Because it’s not going to manage itself, not if you are doing anything remotely complex. And even if you are not doing complex things, DYI is only for techies, cloud or not.
That depends on your definition of 'cloud'. If your app fits into existing PaaS models (Google App Engine, Heroku, et al) then you can dispense with the ops/devops team entirely. Yeah, they're expensive, but nowhere near as expensive as hiring.
You’ll have infrastructure deployment engineers no matter what you do. But you’ll only need infra ops staff (as opposed to application ops staff) if you’re running your own infra.
There’s nobody you pay, in your own company, to sit around watching S3 or Lambda or SQS for degradation. That ops service is part of AWS’s job that you pay for through each service’s fee structure.
Except, TFA talks about renting a server rather than running your own infrastructure. ;) Sure, with AWS there is less work to do (if you have the skills to set it up in the first place, or it’s £££ time) but TFA is right in that you get much worse performance per £.
Comparing it to a full time job is not really fair.
From my experience, doing sysadmin for a single server only takes time initially. You might spend a couple weeks full time to get everything going, but 6 months in it's probably only a couple hours a week.
I’m glad HN doesn’t have downvoting for articles. It’s probably one of the reasons we still have such a good diverse range of article topics, including controversial ones.
Add downvoting and the article queue would quickly become the same groupthink-swamp that the comments section has become.
One of the nice things about HN is you can still find articles that are against the grain and have topics that challenge the comments section’s “one correct opinion”.
I think you’re right. But I think you and the OP are probably coming from different perspectives. Yes if you have frequent scaling issues you probably need the cloud. But a lot of companies shouldn’t have ever have scaling issues and are buying into the cloud to be hip. Just like Hadoop and other big data technologies they didn’t need in 2014.
But how many companies are never going to have scaling issues, and also have the resources to hire a good in-house sysadmin to manage their servers? And sure, maybe you have somebody on staff who can handle the sysadmin duties now, but what if they leave? Do you still have the resources to maintain your own servers?
The author's advice is good for basically one scenario: a personal hobby project run by somebody who already knows they would prefer to run their own servers instead of use a cloud host.
You need good people before you can hire good people. If you don't already have any sysadmins on your staff, how are you supposed to identify the good sysadmin candidates from the bad ones?
I'm sorry who is this person and what authority do they have?
His "homepage" is a raw directory dump with a high-contrast black background (the preferred color scheme of 10x engineers).
Isn't that enough authority for you?
BTW if anyone knows of more nuanced explorations of Cloud v. DIY tradeoffs -- we all know there's a tipping point in favor of the latter, somewhere -- please do share.
>more nuanced explorations of Cloud v. DIY tradeoffs
100% agree. This article goes so far as to be farcical, but that doesn't mean there isn't something worthwhile to be had in the conversation... as long as that conversation was framed appropriately. Especially around cloud value-adds like managed databases and Lambda and security technologies like CloudWatch/Trail.
I'm the sole (part-time) dev-ops resource for a very small company and we (by which I mean I) DIY just about everything running on a small VPS on DigitalOcean. I run my own database and manage my own dokku and user access etc. I use S3 and have dipped my toes into Lambda but I always wonder if I'm missing out on something not using cloud technologies more heavily.
I would absolutely love an analysis of what cloud technology companies SHOULD be using and at which point they become or are no longer worthwhile in favor of an alternative.
I wouldn't go so far as the author when recommending tech stacks for startups or hobby projects. The main benefit of cloud providers is that you don't have to learn and pay attention to all the stuff he lists, and you can focus on building the software you want to build.
But I do think he's performing a useful service in highlighting just how much you're paying for not having to hire a devops or sysadmin person. 10x performance differentials have been replicated not just by him but in other benchmarks. I remember seeing a chart when I was still working at Google and GCP was being justified that showed a graph of hard disk prices vs. S3 rates - since inception (2007-2012 at this time), S3 rates had gone down by a factor of about 2-3x, but price/GB of physical hard disk space had gone down by ~100x. Amazon is hiding all of this improvement in the physical hardware behind vCPUs and opaque billing, and it becomes pure profit to them.
I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins. At that level you could easily afford them, and you probably can get old Linux greybeards cheap now that everyone thinks the job description is obsolete.
because saying your business is built for and in the cloud gets you investors and street cred. further, if you are having a bad quarter, you can squeeze your infra to lower opex. If you built all onprem, you had to invest capex heavily upfront, and you can't slow down your opex very much in the bad times.
> I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins.
I'm not surprised at all: that's thanks to the beauty of lock-in effects. Especially all those cloud-specific managed services make it really hard to switch, and even if you are treating the cloud as a pure VM platform, it is considerable work and risk to switch to a self-managed bare metal setup.
Basically, if you don't start bare-metal (incurring having to assemble the necessary know-how to do so) you will have a hard time to get clean from the cloud drug later on, when you think you're ready to afford having the know-how. I think this aspect has been a main driver behind the linked article.
This is why I'd really like to see projects like Eucalyptus take off, along with (possibly) some standardization of "infrastructure API": https://www.eucalyptus.cloud/index.html
S3 isn’t a hard disk; that’s EBS. S3 is ~17 copies on object-level-mirrored hard disks (allowing a choice of different mirrors per object, preventing overloading and rebuild degradation) attached to a lifecycle event queue, a load-balanced webserver cluster, and an async tape archive.
S3 costs pretty much what that infrastructure costs to run yourself.
But certainly, go ahead and run a Riak CS cluster (plus those other things) on top of some Herzberg servers, and try to outcompete S3 on price. I—and many others—will be waiting. :)
Bandwidth prices vary widely and some are bad, so the “all in” price can be hard to figure. It’s definitely cheap to let your data just sit there. Move it anywhere — it might get expensive.
AWS charges $5 per million write requests, $0.40 per million read requests. Inbound bandwidth is free; outbound bandwidth ranges from free (within same AWS region) to $0.05/GB for public internet at scale.
The latter number can be pricey. But, irrelevant with respect to the failing costs of hard drives.
BTW, I use a wonderful tool called goofys that lets you mount S3 buckets as local volumes. There are a number of constraints (can open for read, or write a file once) but it works well for a wide range of legacy applications.
You might want to highlight that rclone has FUSE support – you have to scroll down the page a bit to learn that your comment wasn't comparing unlike things:
>The main benefit of cloud providers is that you don't have to learn and pay attention to all the stuff he lists, [...] I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins.
$12 million/year for cloud seems like a high enough threshold to go internal but I'm not even sure that's enough money to sway executives away from cloud.
The main issue is that internal IT departments at corporations have slow service from the IT staff compared to AWS/GCP/Azure and its engineers. Corporate IT departments typically don't treat their coworkers in other departments as internal customers. Instead, they treat them as adversaries and a nuisance.
That's why at non-tech corporations, the first group that experimented with the new AWS cloud service were the development teams. They got fed up with IT service backlogs waiting for new dev/sandbox/text servers. They got tired of waiting for IT to requisition a server and then install the os, db, etc. If the programming manager then asked IT to install an Oracle update, the IT department might say "well, Susan our db admin is on vacation till next Monday so we'll get to it then." Those kinds of inter-department interactions were frustrating to the internal customers of the IT department. With AWS, the programmers got their sandbox servers in spun up minutes instead of weeks or months. Once the dev teams were sold, the more risk-averse and mission-critical production workloads eventually moved to AWS too.
The author of this article doesn't seem to understand the dysfunctional relationship between internal IT and their coworkers they serve. It's more about responsiveness and iteration speed of the AWS/GCP/Azure employees' vs the IT employees than expensive AWS EC2 compute power vs cheaper internal racks of cpu.
E.g. The Guardian pays AWS more for cpu+diskspace+network than what they can build on their own. That's the raw hardware costs. But it doesn't matter because Guardian's internal IT staff with an internal cloud stack just can't match AWS: https://www.computerworld.com/article/3427004/the-guardian-g...
To me, it totally makes sense for Dropbox to migrate off of AWS to save money. They're a tech company and have the engineering culture to do it. A lot of non-tech companies (like The Guardian) don't.
> With AWS, the programmers got their sandbox servers in spun up minutes instead of weeks or months.
That's how it starts but not how it ends.
Once you move everything to the cloud, the same IT staff get put in charge of it. Because they have a requirement from somewhere up high to make sure everything has antivirus on it, and there is a Windows domain for the desktops so the servers need to be in it too, and they want to have a test environment with specific versions of specific software matching the ones in production which means you have to use the corporate VM images instead of the AWS ones and have IT go through their VM provisioning checklist for each new instance, etc.
And some of those reasons are bad and some of them aren't, but either way they exist, so if you want to provision something from AWS now it has to go through IT who takes just as long as they ever did to do what amounts to the same thing as before only now it's more expensive.
Counterpoint from someone who did developer support back in the pre-cloud days.
Even pre-virtualization, I could instantiate servers very rapidly with network based imaging. With pre-staged templates of various os/program combinations. Applied them to both lab and production deployment. So that Dev, QA, and Prod could all work from the same base images. With versioning of the images based on patch level.
Then I would subsequently get blamed for every program error by the dev team. With no other justification than "it worked in the lab. Fix your server!" Running a packet analyzer, I would often find things like queries trying to go to a lab db server rather than prod. Or a dependency on an out of date/unpatched/vulnerable version of some supporting software that the development team would then attempt to pressure into production rather than update their code.
For every obstinate sysadmin you've encountered, there is an ops pro who has received a ticket with nothing but a list of compiler errors combined with 'pls fix asap'.
I know the CW is that sysadmin/devops is being automated away, and a lot of people think programming is about automating all the things (or at least as much as you can), but doesn't what you're saying prove that they aren't obsolete for large-scale providers?
I have replicated the performance differentials (for Azure), and they definitely suck.
But people are REALLY REALLY expensive (not only the ongoing costs, but hiring, firing and overhead), and Cloud has another huge advantage: accounting tricks (which help a lot to said midsize firms).
I would bet the vast vast majority of aws customers are not there because of the dynamic resource scaling. Rather, they're there because aws has become the default system provider. At least for the sfbay startup.
I agree, they are very expensive for what you get at any even modest scale. (Except for s3 which is absolutely amazing). However, if you want to run your own stuff, you're buying into knowing the details of how it works. eg you better understand what vacuum is for pg, and how it interacts with availability, and be willing to create backups, etc. AWS does do real work; it's just expensive.
At one of my previous gigs there was a big push to get out of the colo and into the cloud, part of which because of perceived “cost savings” vs running our own hardware. I think most of it was “everyone else is moving to AWS”.
All I know is we never had a $30,000 “oopsie” in the data center because a dev left a bunch of stuff running they should have shut down.
There are many aspects of our business that are in the cloud today that I am confident do not have scaling challenges, would be just fine running out of a single VPS.
We aren't going to suddenly have 20x employees one day, a lot of our infrastructure is highly predicable and stable in load and demand.
Some things need instant scaling, sure. But not everything that's in the cloud does.
The internet worked pretty well without Cloud. I reckon a good 90% of the businesses out there don't need cloud, and all that crap about capacity. The issue is that admin cost is very expensive. To get your own network pipe, have someone wire it up, setup your routers, firewall, do backups, etc. That costs tons of money. So even for a small business that can run fine of a raspberry pi, the knowledge to run onprem is so expensive that it's cheaper for them to pay $1000 a month and go to the cloud than pay network/sysadmin $50k a year.
Their point (in my interpretation) is with bare metal you can get more capacity than you need, for cheaper.
With some cloudflare-ing you could quite easily get away with a single origin server in Germany, too. The latency from a rails app is probably higher anyway.
My anecdata with hetzner - 1.5 decades of lightly used servers for hobby project. One power supply failure resolved in a day. They have some quirks, but.
If you truly have to scale out, sure, cloud is great (sort of). But you can engineer your solution to take the benefit of both, you win.
> Man I wish HN had a downvote option for articles that are just not worth reading.
I have to strongly disagree with this sentiment. Thoughts which are against mainstream deserve to be visible and thus open for debate.
My company for example decided to go with our own production hardware. I'm not saying it's easy. But it's doable. And very educational. =)
One has to remember that running your systems on 3rd party cloud provider will not strip you from responsibility and/or software maintenance cycle which usually presents most of the "human workload". Their nines are not yours.
According to my experience with two reliable machines (redundant PSU's etc) one can create a very reliable service platform. Using correct kind of management tools of course is essential. RAID disk system and a reliable backup arrangement are essential too.
Compared to pricing, features and UX cloud has gotten so much cheaper and easier to use if compared to just a few years ago, let alone the first steps AWS took back in the day.
I use the host he mentioned - they have an API, and my average time to deploy a baremetal physical machine is under 60 seconds via API.
They don't really do custom specs, iirc part of the low pricing is because of how much useful automation they have. Sure, it's not cloud cloud, but if I have a personal side project that I need a few terabytes of ram for, there is no way I can afford AWS/GCP/etc.
The thing is that most baremetal hosts are exactly as the parent described - annoyingly slow, manual, etc.
Hetzner is a little gem here, where they have a large selection of pre-specced machines that are ready for automated deployment, and they also show you a pool of cheaper machines that were returned by other customers/terminated/cancelled and automatically wiped with varying specs that you can also automatically pick out via API (this includes random upgrades that the ex-customers added).
If you manage to fit in the square hole that their automation occupies, it's a great fit.
yeah it works if you only need to serve europe. and if you do not need a private network for your servers (for metal they cost a lot of money, even on hetzner, they have a cloud tough which has a network feature). also keep in mind once you try to scale hetzner, is quickly a bottleneck. > 100 servers per hour.
EDIT: oh and also server buying/renting/installing is probably the smallest problem when running your own metal/colo or renting servers.
a big use case for clouds is, data sharing between hosts via object storage/moveable disks, etc. which is non trivial on "bare metal/colo/renting" and that is just one use case.
There are certainly a lot of cons, yep. I feel like a lot of the arguments here are a bit apples and oranges, some people are getting really upset over it.
But sometimes, you just need a ton of bandwidth, compute, RAM, etc. and don't care about managed services.. I don't think there's a single major cloud provider that will give me a 2TB Intel DC SSD to use 100% of r/w for $20/m. And not being billed per-GB bandwidth is a massive relief for certain use cases.
I can KVM into it with a Lantronix Spider. I have dozens of them, and none of them have shown to be anything other than a physical machine; I can set up RAID myself, all the hardware data is passed through (there are no hypervisor layers in between).
I would be incredibly impressed if it was not baremetal.
I do have one dislike, which is how harshly they enforce mac addresses (so spinning up new VMs and bridging them into the network gets that mac blocked until you register it in the API or panel). And I guess the other dislike is that they're only in Europe, which is 200ms from me.
In many cases, using the cloud is a business and staffing decision more than it is a technical one. It's a decision to accept a monthly cloud bill above a certain threshold instead of maintaining an expensive and perhaps extensive operations staff.
The number of times I've run into this at clients is ridiculous. Either there isn't enough space in the rack, or they're out of network switch ports, even down to being out of SFPs or network cables themselves. I've even seen clients run out of power... as in, they couldn't add one of our servers to the data center because the UPS and power delivery systems could not safely handle the load of one more server.
Running a datacenter is a lot harder and more costly than most people think.
Alongside a host of other issues ("Don't use CDNs because they increase your complexity" is a wild idea coming from someone advocating bare metal boxes) I think there's a confusion of terms when it comes to "cloud"
I prefer renting virtual private servers from Digital Ocean (or, yes, EC2 instances) and configuring them myself. Doing so is perhaps more involved than buying into the AWS/GCP/Azure ecosystem, but I already know the tools involved, and I value the safeguard against vendor lock-in/buy-in enough to spend more time on them.
Those VPSs are "cloud" solutions in my opinion, and a happy medium between AWS and the author's unnecessarily-profanity-laden proposal that everyone should rent/buy metal. You can be "pro-cloud" and "anti-closed-cloud-ecosystem" at the same time.
He means that in his opinion text files are fine for low value analytics until they don’t fit on an SSD you can afford.
In the “roll your own, close to the metal” world this is a pretty sane opinion when the alternative is like, spark or hadoop or something that suck up ops time.
He lost it on "don't use CDN". Asia/Europe/America ... if you run a little bit more world wide you will be thankful for every sort of CDN when it's not just text.
Kind of along the vein of someone telling me I am paying to much for my fancy luxury sports car and completely ignoring the reasons that I find value in it.
165 comments
[ 5.1 ms ] story [ 221 ms ] threadI'm just glad I can do my things without even having to think about, let alone maintain, 1/10th of all this, really. But YMMV, I guess.
Do cloud providers increase your ability to choose scale-up instead of scale-out? Instead of your sharded and replicated MySQL that is almost inoperable and never seems to maintain consistency, would you be better served with one gigantic database? Sure it costs $13k/month to rent one from Google but it costs a quarter million to buy one from Dell. These are all aspects of the decision-making process.
https://www.techempower.com/benchmarks/#section=data-r18&hw=...
And that benefit only improves with scale.
The trick is to make yourself the linchpin, so that once the site grows large enough you can never leave, as you're the only one able to keep that writhing mass of wires up and running.
Doesn't cost a dime to do this, and not more than 20 minutes of work and 10 minutes downtime from my side.
Your work isn't free. Application downtime is costing you.
Renting bare metal instead of owning does not meet this lowest bar. Getting some remote hands service on top with your rental does not change this classification.
Many similar benefits when you have a huge for-profit entity running the infrastructure at their scale. They optimize a lot for you.
AWS almost never forcibly spins down legacy functionality, even if they stop advertising it, unless literally no one is left using it.
That's actually a selling point of AWS to enterprise: while idealistic devs may like “our cloud vendor will force us to use the latest and greatest so we don't have to have that fight”, enterprise buyers don't want “we’re going to be compelled to reengineer our working systems on someone else's timetable”.
That's hard to derive from the same statement - there's a point after which you can afford your own 8 member SRE team across 3 timezones.
The cloud is incredibly attractive when you consider your scale problems as unknown (as a startup, the blitz scale day isn't a good time to go rack nodes yourself) & mostly as way to scale down the costs faster than physical infrastructure + associated human costs can.
I've started looking at this as slightly different model - I'm no longer hiring a good operations manager for those 8 people, I'm hiring a decent programmer who can automate and have code handling deployment tasks instead of a human being taking orders (i.e rack an NVMe + reboot this).
Honestly, even with your physical infrastructure, the trend is towards API based deployment models.
My time at Zynga working on ZCloud, migrating off EC2 to on-prem, that is what I saw. The API based deployment model + reactions from machines (on failure), was worth building everywhere - now that applies to k8s.
Maybe, but you probably have to go pretty far on the cloud maturity spectrum (using cloud native services effectively, not simple VMs like EC2 or even comparatively thin services that abstract just the machine/OS layer like non-Aurora RDS) before you are saving more in local-machine-focused ops teams than you are paying in cloud-focussed ops teams. The latter are more likely to be timeslices of time of team members that are primarily developers, but its still work with a cost.
But yes, this is outsourcing a problem for a cost.
I switched from the dedicated hosting that he advocates to AWS long ago because I was always having things get broken by the people at Softlayer. For instance one time I upgraded my network port and somehow that broke my record in the issue tracking system so I could not longer put issues (or any work reports in.)
There was the clunky and expensive backup server (I'm not sure if it would really restore...) and the breaking point was when somebody added a new disk but they did it wrong so that the partition table got overwritten when the machine rebooted. I was able to fix the partition table but then I moved my files to EC2 as fast as I could because i didn[t have time to deal with that.
As for queues I wonder what is up with that. When I build systems based on message queues they work OK, but I've got an eye for detail and for architecture that many people don't seem to have because I always see other people get in trouble with them.
But for most people, it's simpler to just use them and get it over with, and I'm glad it worked for you.
I've never seen invalidation in decades. you just have unique names.
Yes cloud is expensive in a raw performance per $ sense, but that isn't it's value proposition. Most workloads aren't limited by the processing power that you can buy anyway. The draw for cloud is that it abstracts away entire classes of problems. Certainly it provides new problems as well, but those can be worked on in concert with everyone else working on the same platform.
"Don't go to the cloud, just buy your own servers" completely ignores the reason anyone rents cloud servers in the first place. If I could easily say "I need exactly this much capacity, no more no less, with no unexpected scaling needs and no code/infrastructure changes" then I'd be sitting pretty. Now for a show of hands, how many companies does this describe?
And of those companies that raised their hand, how many can say "I'm fine with just having a server in Germany" who will then go on to say "and I don't need a CDN to serve customers in other regions"?
>keep running live/live setup...This is very critical
Oh I didn't know it was that easy. If all you have to do to keep a server running is say "stay running, it's very critical" then of course no one needs any managed services. Outages are solved forever.
Completely worthless advice from a text file written by... I'm sorry who is this person and what authority do they have?
Also, good luck finding a Dev Ops engineer for $57,600 a year.
So the salary we're really talking about actually is less than 40k a year.
Good luck ^^'
There’s nobody you pay, in your own company, to sit around watching S3 or Lambda or SQS for degradation. That ops service is part of AWS’s job that you pay for through each service’s fee structure.
From my experience, doing sysadmin for a single server only takes time initially. You might spend a couple weeks full time to get everything going, but 6 months in it's probably only a couple hours a week.
Honestly? Lots.
Add downvoting and the article queue would quickly become the same groupthink-swamp that the comments section has become.
One of the nice things about HN is you can still find articles that are against the grain and have topics that challenge the comments section’s “one correct opinion”.
Ya just gotta know your business needs.
The author's advice is good for basically one scenario: a personal hobby project run by somebody who already knows they would prefer to run their own servers instead of use a cloud host.
The ones that know how hire good people. If a company can’t hold on to talent then they’ll probably fail eventually. Cloud won’t fix bad management.
His "homepage" is a raw directory dump with a high-contrast black background (the preferred color scheme of 10x engineers).
Isn't that enough authority for you?
BTW if anyone knows of more nuanced explorations of Cloud v. DIY tradeoffs -- we all know there's a tipping point in favor of the latter, somewhere -- please do share.
100% agree. This article goes so far as to be farcical, but that doesn't mean there isn't something worthwhile to be had in the conversation... as long as that conversation was framed appropriately. Especially around cloud value-adds like managed databases and Lambda and security technologies like CloudWatch/Trail.
I'm the sole (part-time) dev-ops resource for a very small company and we (by which I mean I) DIY just about everything running on a small VPS on DigitalOcean. I run my own database and manage my own dokku and user access etc. I use S3 and have dipped my toes into Lambda but I always wonder if I'm missing out on something not using cloud technologies more heavily.
I would absolutely love an analysis of what cloud technology companies SHOULD be using and at which point they become or are no longer worthwhile in favor of an alternative.
But I do think he's performing a useful service in highlighting just how much you're paying for not having to hire a devops or sysadmin person. 10x performance differentials have been replicated not just by him but in other benchmarks. I remember seeing a chart when I was still working at Google and GCP was being justified that showed a graph of hard disk prices vs. S3 rates - since inception (2007-2012 at this time), S3 rates had gone down by a factor of about 2-3x, but price/GB of physical hard disk space had gone down by ~100x. Amazon is hiding all of this improvement in the physical hardware behind vCPUs and opaque billing, and it becomes pure profit to them.
I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins. At that level you could easily afford them, and you probably can get old Linux greybeards cheap now that everyone thinks the job description is obsolete.
I'm not surprised at all: that's thanks to the beauty of lock-in effects. Especially all those cloud-specific managed services make it really hard to switch, and even if you are treating the cloud as a pure VM platform, it is considerable work and risk to switch to a self-managed bare metal setup.
Basically, if you don't start bare-metal (incurring having to assemble the necessary know-how to do so) you will have a hard time to get clean from the cloud drug later on, when you think you're ready to afford having the know-how. I think this aspect has been a main driver behind the linked article.
S3 costs pretty much what that infrastructure costs to run yourself.
But certainly, go ahead and run a Riak CS cluster (plus those other things) on top of some Herzberg servers, and try to outcompete S3 on price. I—and many others—will be waiting. :)
And that's ignoring the Glacier storage class (which probably didn't show up on that historical graph), which is a fifth of that.
The latter number can be pricey. But, irrelevant with respect to the failing costs of hard drives.
And Glacier Deep Archive is a further fourth of that.
You may be interested in the 'rclone' tool which does all of these things, and more, with all of the cloud providers:
https://rclone.org/
https://rclone.org/commands/rclone_mount/
$12 million/year for cloud seems like a high enough threshold to go internal but I'm not even sure that's enough money to sway executives away from cloud.
The main issue is that internal IT departments at corporations have slow service from the IT staff compared to AWS/GCP/Azure and its engineers. Corporate IT departments typically don't treat their coworkers in other departments as internal customers. Instead, they treat them as adversaries and a nuisance.
That's why at non-tech corporations, the first group that experimented with the new AWS cloud service were the development teams. They got fed up with IT service backlogs waiting for new dev/sandbox/text servers. They got tired of waiting for IT to requisition a server and then install the os, db, etc. If the programming manager then asked IT to install an Oracle update, the IT department might say "well, Susan our db admin is on vacation till next Monday so we'll get to it then." Those kinds of inter-department interactions were frustrating to the internal customers of the IT department. With AWS, the programmers got their sandbox servers in spun up minutes instead of weeks or months. Once the dev teams were sold, the more risk-averse and mission-critical production workloads eventually moved to AWS too.
The author of this article doesn't seem to understand the dysfunctional relationship between internal IT and their coworkers they serve. It's more about responsiveness and iteration speed of the AWS/GCP/Azure employees' vs the IT employees than expensive AWS EC2 compute power vs cheaper internal racks of cpu.
E.g. The Guardian pays AWS more for cpu+diskspace+network than what they can build on their own. That's the raw hardware costs. But it doesn't matter because Guardian's internal IT staff with an internal cloud stack just can't match AWS: https://www.computerworld.com/article/3427004/the-guardian-g...
To me, it totally makes sense for Dropbox to migrate off of AWS to save money. They're a tech company and have the engineering culture to do it. A lot of non-tech companies (like The Guardian) don't.
One deep problem with the "internal customer" model is that you don't get a larger budget when you serve your "internal customers" better.
That's how it starts but not how it ends.
Once you move everything to the cloud, the same IT staff get put in charge of it. Because they have a requirement from somewhere up high to make sure everything has antivirus on it, and there is a Windows domain for the desktops so the servers need to be in it too, and they want to have a test environment with specific versions of specific software matching the ones in production which means you have to use the corporate VM images instead of the AWS ones and have IT go through their VM provisioning checklist for each new instance, etc.
And some of those reasons are bad and some of them aren't, but either way they exist, so if you want to provision something from AWS now it has to go through IT who takes just as long as they ever did to do what amounts to the same thing as before only now it's more expensive.
Even pre-virtualization, I could instantiate servers very rapidly with network based imaging. With pre-staged templates of various os/program combinations. Applied them to both lab and production deployment. So that Dev, QA, and Prod could all work from the same base images. With versioning of the images based on patch level.
Then I would subsequently get blamed for every program error by the dev team. With no other justification than "it worked in the lab. Fix your server!" Running a packet analyzer, I would often find things like queries trying to go to a lab db server rather than prod. Or a dependency on an out of date/unpatched/vulnerable version of some supporting software that the development team would then attempt to pressure into production rather than update their code.
For every obstinate sysadmin you've encountered, there is an ops pro who has received a ticket with nothing but a list of compiler errors combined with 'pls fix asap'.
But people are REALLY REALLY expensive (not only the ongoing costs, but hiring, firing and overhead), and Cloud has another huge advantage: accounting tricks (which help a lot to said midsize firms).
Literally one of the first things they teach you at the Azure training is the difference between CapEx and OpEx: https://docs.microsoft.com/en-us/learn/paths/azure-fundament...
I agree, they are very expensive for what you get at any even modest scale. (Except for s3 which is absolutely amazing). However, if you want to run your own stuff, you're buying into knowing the details of how it works. eg you better understand what vacuum is for pg, and how it interacts with availability, and be willing to create backups, etc. AWS does do real work; it's just expensive.
All I know is we never had a $30,000 “oopsie” in the data center because a dev left a bunch of stuff running they should have shut down.
We aren't going to suddenly have 20x employees one day, a lot of our infrastructure is highly predicable and stable in load and demand.
Some things need instant scaling, sure. But not everything that's in the cloud does.
I think there are valid reasons for each viewpoint.
With some cloudflare-ing you could quite easily get away with a single origin server in Germany, too. The latency from a rails app is probably higher anyway.
My anecdata with hetzner - 1.5 decades of lightly used servers for hobby project. One power supply failure resolved in a day. They have some quirks, but.
If you truly have to scale out, sure, cloud is great (sort of). But you can engineer your solution to take the benefit of both, you win.
I have to strongly disagree with this sentiment. Thoughts which are against mainstream deserve to be visible and thus open for debate.
My company for example decided to go with our own production hardware. I'm not saying it's easy. But it's doable. And very educational. =)
One has to remember that running your systems on 3rd party cloud provider will not strip you from responsibility and/or software maintenance cycle which usually presents most of the "human workload". Their nines are not yours.
According to my experience with two reliable machines (redundant PSU's etc) one can create a very reliable service platform. Using correct kind of management tools of course is essential. RAID disk system and a reliable backup arrangement are essential too.
We used Ganeti (https://en.wikipedia.org/wiki/Ganeti) in our 1st generation setup. The next generation will use our own manager (Deux).
I know you hate their opinion, but also attacking the character is kinda unnecessary. Can we just not?
https://github.com/jackdoe/txt.black/#start-of-content
https://txt.black/~jack/
In other words this is the OP "zulgan"
user: zulgan created: July 11, 2018 karma: 530 about: https://scrambled-eggs.xyz https://baxx.dev https://github.com/jackdoe
This ignores the pay of the extra people you need to pay to build/maintain your infrastructure..
"The Cloud" (presumably AWS/GOOG/AZURE) has become very expensive and very complex.
Want to scale up? 5 weeks minimum. Your opportunity may have been lost by then.
They don't really do custom specs, iirc part of the low pricing is because of how much useful automation they have. Sure, it's not cloud cloud, but if I have a personal side project that I need a few terabytes of ram for, there is no way I can afford AWS/GCP/etc.
The thing is that most baremetal hosts are exactly as the parent described - annoyingly slow, manual, etc.
Hetzner is a little gem here, where they have a large selection of pre-specced machines that are ready for automated deployment, and they also show you a pool of cheaper machines that were returned by other customers/terminated/cancelled and automatically wiped with varying specs that you can also automatically pick out via API (this includes random upgrades that the ex-customers added).
If you manage to fit in the square hole that their automation occupies, it's a great fit.
EDIT: oh and also server buying/renting/installing is probably the smallest problem when running your own metal/colo or renting servers. a big use case for clouds is, data sharing between hosts via object storage/moveable disks, etc. which is non trivial on "bare metal/colo/renting" and that is just one use case.
But sometimes, you just need a ton of bandwidth, compute, RAM, etc. and don't care about managed services.. I don't think there's a single major cloud provider that will give me a 2TB Intel DC SSD to use 100% of r/w for $20/m. And not being billed per-GB bandwidth is a massive relief for certain use cases.
What makes you think it’s a “baremetal physical machine”? This description sounds very cloud-y to me.
I would be incredibly impressed if it was not baremetal.
I do have one dislike, which is how harshly they enforce mac addresses (so spinning up new VMs and bridging them into the network gets that mac blocked until you register it in the API or panel). And I guess the other dislike is that they're only in Europe, which is 200ms from me.
* It takes forever for Dell or HP to ship your stuff.
* Your co-lo can physically ran out of space.
* Your predecessor will inevitable have tons of one-off unversioned changes in prod.
* Repetitive manual work when scaling multi DCs.
* Making unscalable database choices (non-master-master) because you are not in the cloud mindset.
* Don't worry about CDN? AWS S3 comes with CDN automatically.
* Too many more bullet points to mention.
The number of times I've run into this at clients is ridiculous. Either there isn't enough space in the rack, or they're out of network switch ports, even down to being out of SFPs or network cables themselves. I've even seen clients run out of power... as in, they couldn't add one of our servers to the data center because the UPS and power delivery systems could not safely handle the load of one more server.
Running a datacenter is a lot harder and more costly than most people think.
Not a problem for this guy, he uses hetzner's cloud. Did I say cloud, sorry I mean server rental.
I prefer renting virtual private servers from Digital Ocean (or, yes, EC2 instances) and configuring them myself. Doing so is perhaps more involved than buying into the AWS/GCP/Azure ecosystem, but I already know the tools involved, and I value the safeguard against vendor lock-in/buy-in enough to spend more time on them.
Those VPSs are "cloud" solutions in my opinion, and a happy medium between AWS and the author's unnecessarily-profanity-laden proposal that everyone should rent/buy metal. You can be "pro-cloud" and "anti-closed-cloud-ecosystem" at the same time.
In the “roll your own, close to the metal” world this is a pretty sane opinion when the alternative is like, spark or hadoop or something that suck up ops time.
But yeah, sure, the cloud is a piece of shit. Just a beautiful, magical piece of shit.
I would say - run everything in kubernetes (cloud or not cloud). Change your infra layer to use kuberentes objects.
Then you minimise the problem to : where is my kubernetes cluster, on cloud, on prem, I do not care.
Don't go to the cloud you're such a good sysadmin aha