This isn't surprising at all. This industry is incredibly fad driven. Cloud became the fad, and the buzz was that cloud saves money, so everyone goes cloud, and then cloud eventually costs more than the original stuff did.
The same is happening with SaaS. SaaS eliminates the need for in-house IT! Except it doesn't. It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl.
Not following fads and buzz is a huge competitive advantage in this industry. Founders and chief engineers / CTOs / CIOs take note. Just make sure you can explain why you are not using (insert latest buzzword here).
The bottom line is that you should analyze the situation using your work load, your numbers, your culture, etc., and decide what works the best. Sometimes that's managed cloud. Sometimes it's unmanaged cloud. Sometimes it's bare metal. Sometimes it's on-prem. Your mileage will vary.
> Founders and chief engineers / CTOs / CIOs take note. Just make sure you can explain why you are not using (insert latest buzzword here).
1) It takes amazing intestinal fortitude to fight back against the fad tide. And, when things go wrong, your decisions get the blame even if they aren't at fault.
2) As the article points out, cloud is FINE for startups and small companies. If my startup company reaches gigabucks in revenue and I now have to worry about $75 million in cloud spend, I've done my job. And then some.
3) Cloud is often about blame and liability transfer. I don't want the company website getting hacked to be my problem--I want it to be somebody else's problem. I'm willing to pay for that.
Cloud is just a big toolbox of premade tools and resources that you can play with to your hearts content with much of the friction taken out, both technical and bureaucratic.
You pay a higher price for what you use than if you maintain those tools yourself... but you don't have to maintain those tools yourself. It's hardly a buzzword, it's almost defacto standard nowadays.
I think this is an important point. In many cases the benefit of cloud and SaaS is working around your IT department. It's a technical way of going around human management problems.
I feel like this assessment lacks a lot of nuance. I wouldn't really call "cloud" a "fad" as much as an overused option.
> Cloud became the fad, and the buzz was that cloud saves money, so everyone goes cloud
Cloud services _do_ save money for businesses of the appropriate size (meaning, those that can't or shouldn't be focusing on physical servers and networked hardware).
> The same is happening with SaaS. SaaS eliminates the need for in-house IT!
Again, SaaS _does_ eliminate the need for in-house IT _for certain classes of businesses_.
> It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl.
The cost of employing someone to babysit a SaaS product is much lower than the cost to employ someone to build and maintain an equivalent in-house SaaS product. If you're fine with vendor lock you're saving money.
> Not following fads and buzz is a huge competitive advantage in this industry.
Ignoring all nuance and claiming things that are popular are "just fads" is, to me, so much of a competitive disadvantage that it almost certainly outweights any perceived gain.
> [...] paradox: You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it.
> So what can companies do to free themselves from this paradox? As mentioned, we’re not making a case for repatriation one way or the other; rather, we’re pointing out that infrastructure spend should be a first-class metric. What do we mean by this? That companies need to optimize early, often, and, sometimes, also outside the cloud. When you’re building a company at scale, there’s little room for religious dogma.
this isn't a paradox to me, this has always been the case.... big companies can optimize more. They have more non-optimal points due to sheet size alone
As I read it i couldnt help but think the following
Yes, but the benefit of cloud is we only work to optimize those which gain market adoption. For every twilio there may be 100-1000 startups that did not make it, it's a good thing if those were constructed rapidly, tested for product market fit and then turned off without optimizations applied.
Also if cloud adoption actually accelerates building, then it may also aid its adopters if there is a race to product market fit.
It isnt really a paradox at all. Its specifically the opportunity that companies like Oxide will go after. Basically the stage of your business determines the best course of action. Youd be insane to start with large datacenters and high capex for an undefined ROI for most projects. Similarly you dont understand your requirements or workloads yet, which would also affect the efficiency of your architecture.
Whenever a rewrite happens in software there are usually massive performance gains. The same thing happens with infrastructure.
"We show (using relatively conservative assumptions!) that across 50 of the top public software companies currently utilizing cloud infrastructure, an estimated $100B of market value is being lost among them due to cloud impact on margins — relative to running the infrastructure themselves."
That completely overlooks the actual benefit and the reasons driving cloud adoption, which he states early on and then fails to integrate here. The real cost/benefit analysis would take into account the amount of time, money, and opportunity cost saved by using a cloud in development, a critical time that determines whether there'll actually be a profitable company eventually. Optimizing the cloud costs is certainly important, but being able to spin up highly integrated systems on demand offsets a huge amount of time and capital during development.
A takeaway might be that, e.g., AWS, should offer even larger discounts for large scale operations in order to retain mature customers.
Am I alone in not seeing a paradox here? It's no different from a variety of things that make sense at some scales and not at others.
Seems similar to office space - a 5 person company will find the economics of coworking space compelling, a 500 person company will do better with a long-term lease with a commercial landlord, and a 50,000 person company might have their own property management team in-house. That doesn't create a paradox in the commercial real estate market, just different solutions for different needs.
> Am I alone in not seeing a paradox here? It's no different from a variety of things that make sense at some scales and not at others.
Yes, it's not a paradox at all. Use this thing until it doesn't make sense, and then don't use it anymore (or be comfortable with the lack of optimization). This is not a paradox.
I think point guy tried to make is, that ton of big companies, that would/could benefit from in-house solution still go for full cloud based one without thinking about alternatives.
The paradox is that if the coworking space were the cloud, it would just grow with you, and easily house a 50k-strong company, while you'd be paying the same (high) rate per employee.
Why would this happen? Because what the company builds is a large and growing money-making contraption right inside the coworking space, also using parts of the building for critical functions, and the door is intentionally kept small. The only way to move that contraption is to dismantle it and reassamble in a much cheaper, purpose-built hangar, rebuilding some of the critical parts along the way. During all that time, the contraption would stop making money.
That's the beauty of AWS business model: it's a no-brainer for a startup to use it, but the startup grows, more financially efficient infrastructure options become unattainable because of the very high cost of the move. This is the best-executed vendor lock-in I know.
There's also the argument that some of the cloud advantages (such as AWS proprietary services like SQS) simply aren't directly available as selfhosted replacements. You end up needing a whole new team to do HA/scaling for a core service.
There are off the shelf things for, say, S3 and ALB which are entirely workable, but once you start getting into more complicated stuff (like S3's new consistency semantics, or SQS) then you're looking at a whole small company (at a minimum) worth of additional work. It's a non-trivial expansion, even for a large org with lots of money.
You can avoid using these sorts of services to maintain flexibility/independence, but you lose out on their unique benefits. This isn't an accident. It's not like you're going to be able to selfhost an SES clone and get the same kind of deliverability percentages as AWS netblocks, no matter how many engineers you throw at the problem.
There are ways to engineer around this, but it’s going to depend on balancing risk management with business agility. Maybe you use Backblaze for object storage instead of S3. Maybe you use Cloudflare for CDN and Serverless at the edge. Maybe you use Mailgun or another SES competitor. Containers let you use managed k8s off the shelf anywhere. Lots of options for technologists, PMs, or CTOs to pick from.
It can be done. Putting the right abstractions in front of these services makes it possible to migrate and replacements exist. Okay, granted, there will be complexity in self hosted infra but it is possible.
I don't really think anyone with anything less than an 8 figure budget can get the perf/watt of Graviton2. They're literally not available for sale - AWS had them fabbed and then kept all of them to rent out.
Have you tried? Us startuppy mortals can't buy those today, unfortunately. Maybe in a few years, when they trickle down from the people with 8 figure budgets.
Aurora is the one service that got us to migrate to AWS! Honestly it saved us 5 digits per month over another MySQLaaS. For us it was a very good deal. We could’ve dedicated engineering time to reduce db cost on the old provider but Aurora was drop-in awesomeness.
Hardware is difficult, for sure. The managed services / proprietary services is another story.
Having said that, and please keep in mind that I do not have an idea much about hardware so this is a guesswork; more of a question than a statement..., I do think that the Graviton price vs performance might not have a direct benefit outside of AWS. It's cool for AWS because they have less stuff to manage at that scale and it might be cheaper for them to buy and iterate because only they use it. It looks great to the clients because the perception is that the value is better. But AWS runs 80 AZs in 25 regions so it benefits them at that scale.
Realistically, even if one had couple of hundreds of racks, does it matter at that scale?
> There's also the argument that some of the cloud advantages (such as AWS proprietary services like SQS) simply aren't directly available as selfhosted replacements. You end up needing a whole new team to do HA/scaling for a core service.
It's not limited to some. You absolutely need a whole new team (or teams) to handle your infrastructure, your high availability needs, and also your security.
The nifty serverless offerings and other features are just nice-to-haves in comparison to the core infrastructure work that cloud providers put into your system to keep it running.
Just because cloud providers like GCP and AWS and Azure and etc have everything put together to let you setup your whole infra by running a small script that does not mean nothing needs to be done in order for that to work.
SQS is a message queue service. It's a bit weird to claim that as "simply aren't directly available as selfhosted replacements", a message queue is pretty basic stuff.
> I guess they have no one thinking that far ahead
Small businesses don't care because they are too busy surviving.
Medium businesses don't care because why break what works?
Large businesses don't care because their managerial careers promote short-term priorities and marquee "transformation" projects which are buzzword-driven, and cloud is the current one.
Yep, there the ones with stressed out Ops staff running around replacing hard drives at 3 AM and with managers trying to work out how they are going to pass the next security assessment because they don't handle end-to-end encryption or have sufficient controls in place.
My company recently shifted from on prem to cloud. And we aren’t primarily a software company (although, much like most companies, we are increasingly turning into one).
We outsourced our data center needs, and I don’t remember us ever facing these middle of the night massive breakdowns (we had load balancing between a couple of data centers) and the number of ops issues hasn’t reduced a bit since the switchover.
I do think it’s been easier to improve performance for non US customers, because spinning up servers in data centers across the world seems a little easier around the world, but the millions saved by self hosting would have easily paid for someone to spin us up a Singapore and UK colo servers.
If you spent $X on a SAN rack 10 years ago, that same $X now buys you a fully-managed rack with NVMe drives and redundant, hot-swappable everything. Your Ops staff will get an email about a drive or hardware failure and the tracking number for the replacement part that’s already left the depot and can be replaced at their leisure.
You experienced badly run shop. You can have remote hands (if you don't have your own DC, but buy colocation service). You have spare disks in server.
I don't know, how end-to-end encryption is linked to bare metal, those physical servers are several abstraction levels below.
A startup worrying about long term cloud implications is like a hot dog vendor in new York worrying about how his branding will be received in India when his hot dog empire expands there. 99.99% of startups will never have to worry about migrating off cloud, and if they do, they will have succeeded beyond their wildest dreams, and will be once sequential. Nobody is losing sleep over the prospect that their startup will only be worth 22.4 billion in 17 years, but if they eschewed the cloud and built in efficiencies from the get go the company would be worth 23.7 billion. Not to mention the fact that worrying about long term things like that would take focus off the present and in tease chances of failure.
>This seems like a well known trap that people would know how to avoid with a little planning, but I guess they have no one thinking that far ahead
But you see part of the magic trick was Amazon spent years having companies like Garnter proclaim that cloud was cheaper, and nobody could possibly do on-premises IT cheaper because of economies of scale. As a result you've got CIOs everywhere trying to make a name for themselves by driving their company to the cloud to show immense cost savings. They don't have time to be bothered by the actual financial models or real costs. By the time finance realizes what happened they'll be on to the next gig.
It was honestly brilliant, the number of "cloud first" strategies that originated in board rooms filled with people that don't know the first thing about IT is kind of disgusting.
Twitter was on a laptop, then it was in ruby, then it was in java. Companies grow and change, and if your seriously scaling, then your behavior in different environments needs to adjust.
It's simply a strategic planning issue for the CIO.
Have you been part of any large migration? Even if you try to stay “cloud agnostic” (no using Terraform doesn’t count. Each provisioner is cloud specific), everything takes time at scale. Not to mention data migration costs, project management, regression testing, a company of any scale is probably still partially on prem and has all sorts of interconnecting network points with the cloud provider, IAM, etc.
Of course any migration has costs. But if they are higher because of deep entanglements in specific cloud vendor services, that is a vulnerability. It's already demonstrated that the big cloud providers will cut you off with little warning if you run afoul of their political sensibilities. So the only way to use those services is in the most generic way you can, so that migrations don't cost more because of it.
And if you are using cloud just as an overpriced colo, you aren’t really getting any advantage of being in the cloud. You’re going up spend more money hosting all your own infrastructure on VMs, more time on maintenance, and does it add any business value?
Do you not use IAM? Cloud specific infrastructure as code (again TF has cloud specific provisioners), permissions, compliance, training.
The articles mentions many basic advantages of using cloud platforms as simple infrastructure, without entangling your usage with vendor-specific, proprietary services. Scalability is an obvious one.
I’ve seen this parroted so often with anything infrastructure related. Where do you magical people work that your company has managed to somehow find the time to build out infrastructure that you can swap out to another provider on a whim? My assumption is you must already self-host and run everything without using any external providers.
Ok.. so that still doesn't answer my question. How do you get the time and resources allocated to make your entire platform completely agnostic of specific cloud services while still operating in the cloud? Can you name a single company where that's done and they talked about it at all? We're talking about at least tripling developer workloads and general operating costs here and finding developers who can navigate different cloud and on-premise providers for... what?
Even small companies have multiple engineering teams, and each of those teams has particular needs, and each of the engineers, analysts or general users may rely on some service AWS, GCP or some other cloud provider has. Any single person can now break this ideal flexibility requirement if it makes their life easier.
Happens all the time. I personally worked at co that went from aws to gcp and now are moving to azure and know of a few who did the same. So if you can change clouds you could move to onprem presumably if you can find folks who know how and don’t already work for your cloud provider
> It's already demonstrated that the big cloud providers will cut you off with little warning if you run afoul of their political sensibilities. So the only way to use those services is in the most generic way you can, so that migrations don't cost more because of it.
You are stating that you moved from one cloud to another. The question is how you manage to build a company of any size that can just flip a switch and move to a new service provider without needing to transition teams to using new services and that costing money. If we're talking about moving a simple server from AWS to GCP, sure that's no problem. But if you or anyone at your company is actually using a cloud providers services for more than hosting a server, there is no possible way you can just flip a switch and migrate over.
The discount is not as much as you’d think. They’ll do it for some high margin services like storage and gpu (where markup is just ridiculous) but won’t budge on network for some reason. In the end they gon getcha anyway
Oh and btw the models that i had worked out had ~10-12mo breakeven for colo vs cloud including the discount since you’re doing 3y commit anyway. If you’re going with private dc you have to be really large to make it work but if you have large egress charge it becomes instantly worth it (like in dropbox case)
How’s that working for Dropbox? They aren’t exactly a shining example of moving from the cloud and becoming extremely profitable. They still haven’t made a GAAP profit.
Like Jobs said, Dropbox is a feature not a product.
For the same price you pay for Dropbox and 2TB, you can get the full Office 365 suite plus 6TB of storage or the entire GSuite.
And do you have multiple redundant DCs? Are all of your database writes duplicated 6x across three DCs like Aurora?
> How’s that working for Dropbox? They aren’t exactly a shining example of moving from the cloud and becoming extremely profitable. They still haven’t made a GAAP profit.
That’s just silly - dropbox is profitable if you exclude stock comp unlike box inc (who afaik runs in cloud) and also has higher margins (big shocker here). From techincal side i hear it worked out really well for them too.
> Like Jobs said, Dropbox is a feature not a product.
What does that have to do with anything? Not helping your argument at all
> For the same price you pay for Dropbox and 2TB, you can get the full Office 365 suite plus 6TB of storage or the entire GSuite.
Should be painfully obvious to anyone arguing in good faith why that is. Also do you by any chance think google runs gsuite on aws or gcp?
> And do you have multiple redundant DCs? Are all of your database writes duplicated 6x across three DCs like Aurora?
Yes, you run mutiple redunant colos/dcs and peer them. You’re making it sound like rocket science but it really isn’t. If Internet Archive can do it with their resources (not a dig towards their talent at all) so can unicorns flush with vc cash who supposedly hire “the best”
Dropbox is profitable as long as you exclude the fact that they have to pay people…
How many people who are comparing their costs to using a cloud provider are ignoring redundancy?
And I’m sure your average startup has the capability to maintain their own redundant data center. Besides, does it add business value? Would they really be better off hiring multiple infrastructure people or just paying a cloud provider.
It doesn’t matter why it’s the case. People keep bringing up Dropbox as a shining example of moving away from a cloud provider - and still not being profitable. But ignore a little company - Netflix - that did just the opposite.
It seems that the question you are answering, and that I don’t disagree with and am not asking is ‘can you move to a different service provider?’. Not ‘how do you move to a different service provider without incurring costs to transition to different services?’, which is what my question actually is.
The cost is one time and can be relatively contained. Also if you design for redundancy you dont have to do a hard cutoff - just deploy your next region to a dc/different cloud and then gradually drain old ones.
The difference is the bottleneck in skilled cloud practitioners. There aren't that many people who can deploy these systems at scale, and they are best rewarded at the centre of the system. So the system centralizes. Enterprises would love to duplicate AWS, but they don't have the ability to each hire similar calibre talent and keep them engaged on their pet clouds.
I disagree with this characterization as incomplete. In the post-SUN era there were plenty of people who could build and administer a dozen forms of server OS+hardware, but a confluence of factors caused a greater tide to move greater numbers, in another way.
Some commerce did need to scale fast and easily, and AWS provided that with great strength. But I will add that the management and executive branch were constantly and relentlessly seeking to undermine the bargaining power, day-to-day operations authority, and pay of skilled engineers who could do that kind of setup. Not that it was easy or risk-free .. I did see some powerfully strained eyes and nerves on those individuals who built and ran those kind of servers at various times. But do not leave out the manipulative and gaming motives of executive management to dumb-down admin requirements, seize authority via legal means, and seek the lowest wages across borders while doing so ..
> This is the best-executed vendor lock-in I know.
Another good one is microsoft initially tying excel, word, access, etc to windows. And then IIS, SQL Server, etc to NT. An entire generation of businesses and managers got tied to windows this way.
In a way facebook and social media is a form of "vendor lock-in". The network/community itself and your "investment/time" in facebook makes it costly for many to move to another "facebook" or give up facebook altogether. The more time you spend on facebook and the more people you bring onto facebook, the more "locked-in" you are. It's quite ingenious and I suspect facebook understood this very early on.
The former is lock-in.
The later example is better described as "network effects", i.e. Facebook is compelling to you because all your friends are on there, and the more of your group that engages on the platform the more valuable it is to you and the harder it is to get everyone to jump in mass to another platform.
Having taken a large organization's entire stack the other way in 2016 - from dedicated metal colocation to AWS, after a series of devastating ddos attacks on our datacenter - I've remained cautious and judicious in how I use cloud services, to attempt to prevent lock-in, for this reason. I build all critical systems (including outbound mail - which requires negotiating with Amazon) on raw EC2 VMs, allocated drives and RDS images, and more or less balance workloads between them manually. This involves a bit more hassle than e.g. putting frontends on elastic beanstalk with automatic load balancing, or using buckets or any of the nicer cloud features... I basically treat AWS as a colocation facility where I don't need to get anyone on the phone to reboot a machine at 3am. I've been told I'm "misusing" or even "misunderstanding the purpose of" the wonderful services provided by AWS. But for me, the upshot is that I could probably port the entire infrastructure to any other server farm within a few hours if necessary.
I like how some of the services are very cheap (S3, Lambda), but then at some point you end up using a bunch of other complements, one of which is very expensive and makes the profit.
At which point, as you say, the cost and complexity of a move is huge so you are stuck with a recurring bill.
Another interesting thing is that you pay for 100% of a VM virtual CPU even if it is only utilised 40%, but the cloud is selling your other 60% capacity to another user.
They have no incentive to charge you “based on CPU usage” for VM’s that are always on, even though it is possible. I think this would be a better approach than Lambda - to dynamically add more cores to your OS as you need them.
> Another interesting thing is that you pay for 100% of a VM virtual CPU even if it is only utilised 40%, but the cloud is selling your other 60% capacity to another user
AWS don't overprovision CPUs outside of the t instance range, in which you get vCPU credits, not full vCPUs. If you rent an apartment but don't use it, is it the landlord's fault you don't? The landlord can't rent the same appartement to another client.
Especially on AWS/GCP/Azure, where there are many alternatives ( Lambda, Google Cloud Run, Google App Engine, etc.) where you pay per compute used.
At the hyper visor level, I was under the impression that many VM’s are scheduled onto the same hardware CPU core.
One hardware core can be running 5 VMs at 20% capacity, and each VM owner will pay for 100% of their CPU capacity right? So in this case the landlord is renting your unused apartment to other customers then evicting them faster than you can open the door.
My point is cloud is benefiting from that ability and not passing it on the customers, which they could.
That statement is counter to how "vCPUs" related to "CPU" since early cloud implementation. AWS not taking advantage of the much more efficient "oversubscription ratio" versus thread pinning - unlikely.
The solution is an internal marketplace. Large enterprises formed from conglomeration often flub this. They often end up with a half dozen or more datacenters and ops groups. What do they do?
Consolidate them, and a knife fight happens between middle management machiavellis and the one that wins is not necessarily the best on price or features. It's the one who is best at winning reporting hierarchy wars.
Instead they should pit these groups against each other as internal options for hosting to various services that provide what is needed for growth, adaptation, or cost.
AWS should simply be one of those options. Really it can be the prototyping phase, and then budget squeezing will move the app to other (hopefully cheaper) inhouse options.
What is frustrating from my years in the big corps is that AWS was cheaper than the internal "clouds" or other relationships.
The paradox is that in spite of the huge costs, very large companies are still moving to (or using) the cloud.
I've had this discussion many times where I work, and we've refrained from using cloud services since they are extraordinary expensive compared to what we can do internally.
Yet, people seem to be convinced that prices are ok, and that going outside will let them reduce their infrastructure footprint.
For some reason I don't quite understand, people tend to be attracted by the cloud, even when it makes no sense economically whatsoever, and talking sense into them is surprisingly difficult.
So yes, it seems obvious that people will choose the most cost effective solution. However, it seems like they don't!
>The paradox is that in spite of the huge costs, very large companies are still moving to (or using) the cloud.
In my experience when a large corporation moves to the cloud it has less to do with pricing and more to do flexibility. It's easier to get a budget from IT and do whatever you want in cloud, then to have to wait weeks/months putting in a PO for hardware, getting access to those machines and installing what you want.
This is true. Where I work a PO was issued several weeks ago for a rack of servers, and delivery will be in about 3 months. If cost were no object, those resouces could be provisioned with a cloud provider today. And getting from requirements to RFQ to PO took another 3-4 months.
People are attracted to it because owning stuff sucks. Renting stuff is more expensive but sucks less. This is why businesses often prefer opex to capex even if it is pretty significantly more expensive. It isn't just an IT phenomenon.
Yep people will pay through the nose to move from capex to opex. It baffled me at first, I think it baffles me less (but still a bit) now. Maybe one day I'll understand.
Two reasons: 1. For the investment in the land the house sits on (the house itself is a depreciating asset, but the land may be a good investment), and 2. For the ability to control their own destiny, to be able to make improvements and to have a place to live that does not rely on a landlord's willingness to continue signing leases with you. Note that #2 is totally analogous to owning your own server infrastructure. You definitely give up control when you rent rather than buy compute. But I think people are more sensitive to this in their personal lives than businesses (especially small businesses); after all, you're stuck with your personal life, you can't sell it or declare bankruptcy or just shut it down if things don't work out.
It's an accounting phenomenon: Deducting expenses reduces income tax, which is levied on net income. It is also beneficial when considering the time value of money – money available at the present time is worth more than in the future due to its earning capacity.
Some companies have terrible IT outsourcing and procurement practices such that getting something comparable to aws infrastructure is effectively not possible, as it would require massive internal change whereas moving to the cloud is easy to get buy in.
Also regular devs don't realize but frequently internal IT departments charge obscene amounts of money for managed services.
Devs look at Digital Ocean or Hetzner VM costing $10 for a ton of RAM, storage and bandwidth when the same thing internally in a bank or other big enterprise can cost $100-150. AND be delivered in 3 months, if it's ever delivered.
Absolutely, bargaining for cheap service fees from these suppliers, only to be charged greatly for simple hardware and anything outside the standard list.
> Yet, people seem to be convinced that prices are ok, and that going outside will let them reduce their infrastructure footprint.
That's where you get it all wrong. It is not about the price, at all. It is all about avoiding large upfront capex with a small periodical opex, and in the process have virtually boundless growth potential.
Think about it: when you happen to need a bit more computational resources to run an app, is it easier to convince your boss the need to pay, say, 100€/month for extra VMs, or is it easier to convince your boss to shelve $2k to buy an extra rack? And how many times are you willing to have that same conversation with your boss whenever you run out of computational resources?
> Aren’t the real figures more like $200 vs. $700?
No, not really. You can hardly find old used servers for less than 1000$, and you need to use beyond a a1.metal instance with 16vCPU and 32GB of RAM to even come close to that cost
> Cloud stuff only really makes sense when your resource requirements as super spiky.
No, not really. Cloud stuff makes all the sense in the world if all you need is a VM connected to the internet working 24h/day without problems. You can get that for less than 5$ a month, which would require 3 or 4 years to pay back if instead you waste your money on hardware.
> You can hardly find old used servers for less than 1000$
I picked up a second hand Dell PowerEdge 610 with 96GB of RAM for $400 AU, it's overkill by a large factor for what I need but damn. Either I was super lucky or servers aren't (weren't?) expensive.
> You can get that for less than 5$ a month
pocket change and host it
Until you actually use it and then it ramps up rapidly to the point where you'd be far better off just buying a server.
Good luck getting spares, patched BIOS/firmware, patched DRAC, or anything else resembling support for that old Dell in a timely fashion.
And good luck selling to enterprises when the vulnerability scanner lights up like a Christmas tree with critical and un-patchable flaws on your switches/routers/UPS/PDUs/storage arrays/servers/door locks/cooling system/fire suppression/cameras. Double for the DR site. Because you need all that stuff, and you have to maintain it all 24x7 which means lots of salaries.
The same reason big companies pay premium for external consultants instead of directly hiring.
You don't want to deal with employees, as well as you don't want to deal with HW, data center, and all that comes with it. You know you are paying premium, but you are also delegating tons of headache and responsibilities.
The mining companies I work with often hire equipment semi-permanently instead of buying it. It’s an insane cash crop for the hire companies because the payback time on the equipment is a few months and then it’s pure profit, but apparently a $700/week car rental payment is easier to get through accounts than a one off $50k purchase.
Well I guess by renting they can get the latest and greatest machine available instead of dealing with an old machine bought say 5 years prior. If the machine breaks, well you just get another one. No maintenance cost for the company.
I mean it depends. Buy it is not always the best thing to do. Renting may be more convenient sometime.
You'd hope so but these are the same machines on site that have been there for years and they're still paying the same per week for them. It literally just comes down to the capex/opex dilemma, individual caps on expenditure without review, and lack of motivation to improve efficiency if it doesn't land in their pocket from everyone else.
Running in house, you need to know what you want, and you need to be pretty accurate in your estimate. And you need to pay today for what you're going to need in six months, a year, three years.
And if you're wrong about those estimates, the costs are real. Overinvested in hardware upfront and your project will never break even. Underinvest, and when the marketing campaign kicks in and you get flattened on cybermonday, there is no limit to how much revenue you might miss out on.
I'm willing to pay more money to reduce the downside of my being wrong. Cloud services offer so much flexibility that I don't even have to make guesses about some things beyond my information horizon.
On prem might be cheaper but it is far from an equivalent service.
You don’t need to be particularly accurate. The cloud is so expensive you have plenty of wiggle room while saving money. Worst case you can still use that hardware on another project. Going the other way adding more hardware 6 months out is generally trivial.
A paradox is only at the most coarse-grained level. Cloud makes sense because of its cost-efficiency in scaling from nothing upwards. Then fails because of cost involved in keeping it running at scale. The thing to recognize is that what cloud is good for is rapid change. Once you have something stable, even at not-huge scales, bare-metal hosting may still make more sense.
All this happens willingly unless growth happened with your eyes closed, or really had no choice keeping up with it. Sure, use the cloud, but don't use every cloud-proprietary convenience. Use opensource software in a near-standard way. Even if using the cloud-vendor's offering, refrain from using proprietary extensions. If your plan is to grow and get out before things stabilize, then that's on who's come onboard, not resisted the conveniences, and remaining.
The paradox is the sunken cost of your product easily becoming coupled to the cloud.
Moving office is relatively simple, you call a moving company to lift all your chairs, print new business cards, hire a bigger maintenance crew and it's done within a week.
Moving cloud could in worst case mean a 6 month complete rewrite of your core product depending on how much dependent you are to CosmosDB, Lambda or other shiny AWS stuff.
You could say buying a machine you previously rented is the same thing but when looking at code migrations, rewrites and launching new infrastructure cost time from R&D, not just money.
Disclaimer: I'm a Co-Founder at Vantage, a cloud cost platform. I also worked in public cloud for ~6 years at AWS, DigitalOcean, etc.
While repatriation can make sense at a larger scale company, startups and SMBs can yield the same benefits discussed in this post by simply tracking and optimizing cloud spend.
We try to make this as easy as possible for people with https://www.vantage.sh/ - where we're already helping thousands of individuals, startups, SMBs and enterprises as it relates to AWS.
In 2012, I was working on a migration from EC2 to on-prem & was working on infrastructure building for zCloud. The cost factor was huge, the switch out of EC2 literally paid for itself in a lot of ways.
The work was very interesting, because a lot of it was actually building a private cloud for internal customers & the work primarily centered around a virtualized data-center aimed at boom-bust cycles of games (15+ million users for 6 weeks, drop to 2 million for a month and down to a million in another week).
The issue is that the infrastructure cost is somewhat constant when dealing with that sort of fluctuations in revenues, so the cost to revenue ratio was unpredictable (while the cost was).
So what happened in the end looked a lot like a fire-sale of hardware when the cost was unbearable, while if it was an end-user cloud, that low-demand phase would be able to cut losses as a spot instance or something.
Anyway, a few years after I left, back to EC2 it is[1].
Why is the infrastructure cost constant? I would have thought that for gaming, at least compute and transit would scale linearly with traffic. If games have a boom bust cycle, aren't they the perfect use case for public clouds?
The infra was on-prem, so it was planned in advance and ops folks who had on-call rotations every day etc & that was rented out by the hour to the game teams (a chargeback model).
The salaries and hardware cost was paid for even if the games had a bust.
The period where it worked well, the games had boom-bust in somewhat controlled fashion where farmville -> cityville -> frontierville -> fishville etc, the traffic would move around rather than die down entirely.
The world turned mobile-heavy and that whole pipeline fell apart while they were restructuring into mobile games (words with friends etc), when the hardware had to be sold or the payments would start to hurt.
At around the same time I was working at a startup in London. We had two sizeable clusters - Hadoop and Cassandra - neither of which was conveniently elastic, but were located on EC2. We had sundry other instances, but those were the major culprits for our ~ £25,000 / month.
It took me about a day to scope out a BOM that would well surpass that (maybe a year's expected growth) of whitebox server gear, and get some quotes from nearby co-los.
IIRC hardware capex was about £15,000, and monthly rack + network something around £3,000. We relocated within a few weeks.
One small bonus was predictable & consistent performance -- back then, the EU-west EC2 offerings were extremely sensitive to noisy neighbours, and our benchmarks never gave anywhere near the same results twice.
I think for genuinely elastic loads, or if you're really addicted to some vendor-only services, it certainly makes sense. I suspect most customers are overly optimistic about how elastic their requirements are, and their stack can be.
I've had the opposite experience where others didn't see the cost savings, and also performance improvements, as a benefit. Some people get "scared" when you don't put things in the cloud. Meanwhile a last gen server with new SSDs running your applications on bare metal can outperform most EC2 instances and whatever networking cross connects at your local DC are available will get you higher bandwidth, lower latency, and more throughput for way less than AWS.
This has been widely known for many years. The crossover point happens much sooner than I think people intuit but most companies never really measure or model it. My experience at a few different companies is that DIY infrastructure pencils out at 30-40% of the cost of the cloud.
There is a learned helplessness when it comes to companies running their own data centers that has become widespread over the last decade. What used to be a fairly mechanical process has almost been mythologized as some kind of arcane art beyond the technical ability of any company that isn't Google or Amazon. Designing data center builds isn't difficult, it is a pretty straightforward albeit detail-oriented blue-collar engineering skill, but it seems few people learn it anymore.
Yea pretty soon we’ll have infra engs that had gone through their whole career without ever working a real server rack. I personally find that thought discomforting
How many more people does it take to manage on-prem (or dedicated or colo) versus managing AWS? For many businesses, the answer is "less than the cost of AWS".
The flip side of this is very large enterprises with fleets of thousands to tens of thousands of servers and tech staffs in the hundreds to thousands that with 50 years experience self-hosting, still manage to TCO some 2x - 10x above AWS / Azure retail list pricing, while also taking 9 months to provision a VM on a server.
If it needs to be said, this is more or less exactly the thesis behind Oxide[0], and matches what we are seeing in the market. It's certainly validating to see a VC firm echo our pitch deck back to us, even if one that (in)famously doesn't believe in hardware! ;)
Yes, but what I'm not seeing in their analysis of variable-cost cloud vs cheaper fixed-cost colo is the huge fixed cost of the personnel to manage it. Yes, I agree that switching my food-delivery business from calling for an Uber whenever I get an order to buying my own car can be much cheaper, but if I leave out the cost of hiring a full-time driver for my new car, my analysis is...incomplete.
Your pitch emphasizes features that make servers easier to manage, presumably lowering (but not eliminating) the cost of personnel, so you're obviously aware of this, but I'm not seeing any "net of additional, fully-loaded personnel costs" in their analysis.
It should still be cheaper above a certain scale, but the breakeven where it will make sense to "repatriate infrastructure" is much higher if you include hiring people, and will go even higher if the cloud providers run the numbers and match most of the savings with scale-based price breaks.
I don't know how every company does it. But I led the datacenter and networking teams at Square from 2011 to 2017. In that time we went from 1 DC cage with 4 server racks to 4 US DCs, one in a Japan and a couple of network pops. A bit more than 100 server racks total.
My team had 2 guys doing SiteOps. They would travel to the various DCs in the Bay Area and Virginia and do all the maintenance, new installs, etc. And sometimes we'd lean on the colocation remote hands to do a few things.
We had about 5 network engineers, that also handled the corporate network. (12 offices and a network backbone that connected east and west coast DCs, offices, etc).
And maybe 2 SWEs who handled things like our host OS install system, etc. Basically the next layer above the hardware.
So 9 people that were required to run all of that stuff. But really, NetEng ended up spending like 70% of their time on corporate network things because we'd add offices faster than datacenters.
So if we focused on production only, we really needed about 6 or 7 people total.
I did the math a few times (every single year) and compared our costs, including people, to the costs of moving to AWS 3 year reserve instances.
Doing it ourselves was always half the price.
Of course the difference here was we built on-prem from the start. So there was no repatriation that had to happen.
Since then, I've been responsible for large cloud infra on all 3 major providers and learned a lot about what kind of discounts you can get when you're in the double digit millions in annual spend.
I still think, at a scale of single digit thousands of servers, you'd be cheaper on-prem, fully loaded. But admittedly, I haven't run the numbers since 2017.
The problem is not pure cost with a competent team: It's that many companies can't even tell that the team they have is competent. And when the team is not competent, prices, and opportunity costs, are so much higher than what you describe. Imagine a world where adding the equivalent of a 4 core VM with 8 gigs of ram is going to take 6 months from the time a team say they need it, until it's actually available. The meetings, the arguments and other minutiae (like trying to get things installed on the server, because of course you need to play telephone games to get a single command run in a server) make the costs of the server look small.
And those companies are in situations where either people have no idea that their teams are performing badly, or lack the political will to do something about it. Adding the services in AWS is just easier, and might be worth every penny.
When you go to a high performing firm and explain the procedures and decisions of a low performing one, people think you must be lying, but it's true: Broken IT departments are everywhere in enterprise companies, and paying double of what a good team would take to do the right thing is a bargain, given that they can't even get a whiff at a good team.
> Imagine a world where adding the equivalent of a 4 core VM with 8 gigs of ram is going to take 6 months from the time a team say they need it, until it's actually available
If this is the case then the CTO knows that something is very wrong.
I don't think your scenario of companies not knowing whether their teams are incompetent makes sense. You can apply this logic applies to all functions.
Except you miss out on all the opportunity cost of cloud-native solutions that get you moving fast for the sake of a complex convoluted mess that you need to hire subject matter experts for to wrangle, instead of focusing on your business logic.
Kubernetes only makes sense from Google's perspective to sell you the managed version "from the creators of Kubernetes!" once you get tired of trying to wrestle it into submission.
I don't think you need to "wrestle it into submission". There are many CNCF projects that work great on bare metal instances like Rook and other companies providing managed DB solutions for your hardware like KubeDB.
K8s isn't that complex from a cluster operators perspective if you've managed bare metal Linux systems before. Set up some PXE + auto provisioning and you can make your node enrollment process:
1. Buy hardware from an OEM.
2. OEM ships you a piece of hardware.
3. Remote hands plug in hardware.
4. Machine PXEs, boots, and is turned into a worker on your cluster.
Canonical (MaaS), VMware (vSphere), and Rancher Labs (Rancher) all have products aimed at running Kube on metal in your own datacenter.
So, essentially, if you start your startup off using Kube in a managed cluster you can, at some point in time later, directly move your code off the cloud and onto your own machines for cost saving. You just need to figure out how to run your DBs and LBs but that's a solved problem as mentioned above.
k8s lets us move fast - we can easily deploy new services in no time. We can also run the entire stack on our desktops in k8s, or have our laptops be a node in a mostly cloud k8s dev deployment and debug services locally.
The opportunity cost is staggering. Not just of the people that do the work, but the cost of focus and top-level priorities over the years it takes to do the work.
Everybody loves to cite DropBox here. The greater arc of DropBox is their product stalled, they lost the enterprise of their market to Box, and they found themselves a commodity in a commodity market. Heck, maybe they'll go back to Amazon if they keep doing things like this: https://aws.amazon.com/solutions/case-studies/dropbox-s3/
But that's not all. It takes a top-level directive to repatriate an entire SaaS. That's at the cost of other top-level projects. It's wild to me that any company that has significant fuel left in the tank would buy back single-digit COGS percentages instead of investing in product that could add double-digit growth for several years at scale.
The only question worth asking right there. It's the story of so many companies. We were doing something incredibly stupid, and in this five-part blog series we describe a slightly different implementation of an equally stupid idea.
The most powerful tools we have are aggregated reports and data retention periods, the same since the 1970s.
Running 10 copies of the same report (but each one with slightly different bugs) across 34 PB of data for 10 different departments is absurd, yet here we are.
Specifically for Dropbox, they're not a retailer like Safeway with 10,000 products and 100's of locations to crunch product and shipping data for - Dropbox could use a single server for their entire digital products data warehouse.
(I operated a data warehouse myself (literally) at Yahoo - 10 servers generating canned web reports for 500 managers. I heard they replaced it with a 1,000 node Hadoop cluster later.)
This makes me wonder whether 34PB (+1PB per month) of analytics data creates more value than it costs to store and process all of it. I’d think that storing only aggregated information would provide nearly as much business insight to make strategic decisions at a fraction of the cost.
What I know from talking to former Dropbox insiders is Drew really wanted to be running a consumer company. He really resisted a push to SMB and Enterprise, even though, at least to me, that seems like the obviously better business.
So not sure if it's so much that the product stalled, but that leadership actively didn't want to move in that direction for too long, then lost the advantage.
Instead they spent a bunch of energy on Carousel and Paper. (I don't have any idea how much of their eng team was working on those things)
slightly tangential, but Box, a storage company that did take the business route, is about 10% more valuable than it was at its IPO in early 2015. That is pretty bad in comparison to most other public software firms that grow like weeds.
Carousel team was huge. My roommates was at Dropbox and said it was 100+ while he had a handful of engineers supporting him for enterprise features. Mailbox was another consumer investment that was largely a waste. Drew really didn't like enterprise.
Dropbox also clearly had a business where what they sold was fundamentally storage and bandwidth. So it makes a lot of difference to their fundamental cost structure how much they are paying for storage and bandwidth.
Most businesses who are operating in the cloud are not so directly purchasing a cloud resource, layering a little value on top, and reselling it; most businesses make their money selling something else, and using cloud compute/storage/bandwidth/infrastructure is not a transactional cost.
For every gigabyte of storage space dropbox sells, they have to buy a gigabyte of storage from someone. And if you're looking ahead to the future of selling lots more gigabytes, you're quite motivated to find a cheaper way to buy them.
So I am never convinced by the argument that because Dropbox left the cloud, that means other businesses should.
"buy back single-digit COGS percentages instead of investing in product that could add double-digit growth"
Its good that we have people focused on lowering cost and optimisation, thats why people in India can afford a smartphone.
On the other hand, hisotry is filled with features and products that never found their niche, from random dropbox features to thousands of products google created and killed
This article only looks at "seen" costs, and assumes that there are no "unseen" costs to running on-prem. Many companies do not have the operational maturity to run on-prem well. The result: high cost of operations, low availability, and large increase in time-to-value.
Second unseen cost: everybody becomes their own SI. So far nobody really sells the "whole stack" for running on prem. I mean hardware, network, virtualization, application, traffic mgmt, etc. I have to buy stuff from two dozen different vendors and cobble it together into a high-labor, rickety Jenga tower of stuff.
These costs are not unseen. They are straightforward ordinary costs and fully accounted for. It only seems expensive and complicated from the perspective of someone that does not have experience with it. To someone that has done it many times before it is relatively simple and almost completely mechanical to get an excellent result. It isn't rocket science, just domain expertise around things like physical infrastructure planning and supply chains that a software developer may not have encountered before but could easily learn.
If a company was going to run their own data centers, they would presumably hire someone that knows what they are doing to lead the effort instead of trying to do it by trial and error.
Also in my experience, on-prem infra, on relatively cheap gear like Dell servers is highly reliable.
Yes, you have to plan, yes you have long lead times to get new gear, DC space, etc. But once it's running, failure rates are pretty low. At least in the single digit thousands and servers.
I've had to deal with much more frequent and odd types of failures on cloud infra than with on-prem.
My experience matches yours. Bulk inexpensive hardware that you can buy by the pallet is consistently very reliable over several years in my experience. I've had more reliability issues with gold-plated "enterprise" hardware and cloud services in practice.
Supply chain management and coordination is, IMHO, the most difficult part of it and is often overlooked in these discussions. Herding vendors to deliver hardware on your timeline from overseas supply chains doesn't always turn out the way you want, even with the best planning.
> It only seems expensive and complicated from the perspective of someone that does not have experience with it.
Isn't that the target audience for this sort of change?
I mean, do you see teams of networking , siteops, high availability, and security engineers hanging around doing nothing and just waiting for a company to decide to go in-house?
No, because those teams do not exist in free-range. That's something a company needs to build and train and experiment from the start until they are able to learn all the lessons.
There's also the unknown of cloud pricing in 5 years. Unless there's a monopoly, at some point, there should be at least one cloud provider who can take advantage of economies of scale and beat the cost of colocation, even with lots of machines.
This analysis completely skips over the "elastic" aspect of cloud infrastructure, which was a key motivator for using these providers since the very beginning and is as relevant today regardless of company size.
My company (which is in fact part of the charts in the article) had every single metric across the board spike 15-20x basically overnight when the pandemic started last year in March. Our entire infrastructure burden was clicking a few buttons on the AWS console and making sure everything was provisioning and scaling as needed. If we had to send out people to buy hard drives and server racks at that time, there is no chance we would have been able to meet the extra demand.
Plus, if you give me a few dozen capable engineers today I'm not going to waste their efforts on rebuilding AWS to get a best-case few percentage point return on our cloud spend. I'll launch a new product for our customers instead.
That’s a valid point. I don’t want to argue pedantry but I’ll say I worked some media companies that experience 5x normal volumes a few times a year during events before returning to “normal”, making AWS an obvious choice.
I’ve also worked for a place so large in AWS we hit walls where we hit Amazon’s literal physical limit and had wait for them to go out and buy the hardware for us to provision.
For Dropbox specifically I don’t see them being any form of special case, if it saves money and they obviously had operational experience so it makes sense
In my experience it's pretty common to stretch your infra out much longer than 18-24 months. Often finance has a 3 year depreciation schedule, and depending on your needs, you can keep existing workloads on existing hardware for many more years.
The last time I ran physical infra, we had sever racks that I'd originally bought, running in production 5+ years later.
Once you're past the depreciation schedule, they are basically free except for power and space. Yeah, at some point and scale, it can make sense to get new gear that is more power and space efficient to pack more into the same power and cooling budget.
AWS still lets me launch c1-c6 instance types. So they still have the older generations sticking around. Yeah, the newer ones are usually more cost effective, but you do have to do work to migrate to them.
Given that interactions to the cloud are through a relatively small set of known APIs, I'm surprised that there isn't there a service which replicates the APIs but with the endpoints being on "repatriated" hardware.
Well, the thing is they're not all that similar (except maybe s3 which has been copied by competition). So terraform or similar abstractions (from pulumi to chef/ansible) are what you get. Or k8s.
APIs are not what's hard IMO. Creating a layer simulating S3 API is pretty straightforward.
Designing and implementing a highly available and resilient storage solution on-prem is much more harder task that will take 99% of time & budget to do properly compared to a compatibility layer.
These people don't see the real value proposition of the cloud. The value is not in "scaling when you need to". Sure, that can be very handy. But that is not what 99% of companies are getting out of the cloud 99% of the time.
If you self-manage, your capital investment is initially higher, and lower over time. At the same time, the effort it takes to reach the same results is always higher, and the quality of the end product may be lower, depending on how much service quality affects your product.
If you pay for managed services, your initial investment is lower, and higher over time. But at the same time, you require less effort, and you get higher quality outcomes.
This is obvious to anyone who has worked in the industry and done both. First, host your own service: JFrog Artifactory, Atlassian Confluence, GitLab, whatever. Now rapidly increase the demands on this service. As demand rises, quality will decrease, because it takes a lot of time, effort and expertise to build a very reliable hosted service. Now switch to a managed cloud instance. Suddenly, the service's average quality increases. Performance is steady regardless of increase in use. There are virtually no interruptions to your product or development.
The impact of a service's quality and reliability has ripple effects. If poor service quality slows down development, that means development quality will go down as people cut corners to try and meet deadlines. If the service is used for production, it means your product's quality will suffer, and that effects your bottom line. So a huge amount of the actual cost is not just paying for a service, but also how much business value is generated or lost due to service quality.
There is simply no way to replicate a managed service without becoming a managed service provider yourself. You have to become a whole new business within a business. It's like a yogurt company also becoming a dairy farm. Running a farm is not easy, and you will screw it up for several years. Seems obvious for farming, but for some reason people always underestimate this when it comes to technology.
On paper, the Cloud's value proposition is scalability. But in practice, the true value is actually as a force-multiplier for your product's quality, reliability, and time to market. (Time to market not just being "I launched my startup" but also "I released this new feature before my competitor")
Having done both, I think I mostly agree with you. And I'm a big proponent of having your engineering team focus their efforts on building and maintaining the product you are selling to customers.
Avoid doing undifferentiated work. Outsource it when you can. Focus on the core product. I think this is especially true as you're scaling a company in the current SWE labor environment. If your business is growing quickly, you probably have money, and can get more funding when you need it. But hiring more SWEs is always a challenge. That is your scarce resource. Don't waste it on undifferentiated work.
All that said, at larger scale and more mature businesses. I think the core thesis of the article makes sense. You may be in the slower growth part of the S curve, but you can still increase your company value and profitability substantially by increasing margins. One way do do this is to invest in cheaper infrastructure.
Then you're taking on some undifferentiated work, but in service of better margins.
But you have to be very careful where that cheaper infrastructure is used, due to how difficult it can be to manage, and its potential impact on the business.
I've worked for a handful of very large businesses (10s of thousands of employees, profits in the billions) running infrastructure and services. With one exception, they all sucked at managing their own hosting. The only one that did well literally poached all the core employees of a web hosting company, and then listened to them.
Except for that one company, none of the others was willing to build their own internal managed hosting company. They all didn't staff right, didn't fund right, didn't do support right, and their org structure and financing model had no capacity for a single department to straddle every BU. (Well, ok, "IT" often did, but "IT" wasn't directly managing and running the underpinnings of every product in the company)
In fact, some of the largest, most well known companies in the US actually moved to the cloud specifically because they sucked so bad at running their own shop that they needed to stem the bleeding by just buying Cloud services that worked. And that's after they hired consulting/management companies to try to jointly manage their self-built setups.
The article makes sense. The idea that larger businesses can figure out how to do it right at scale, sounds logical. But my whole career is one long proof that those theories don't match reality. In fact, I would go so far to say that the larger the organization is, the harder it is to self-manage. The only general category of company that I think is OK to self-manage hosting is one where either almost all of the profit is coming from keeping hosting costs down, or the business value is not impacted by hosting quality at all. I think that's a very small proportion of tech businesses today.
I think they understated the importance and difficulty of retaining a sufficiently redundant skilled workforce to manage the equivalent cloud infrastructure.
214 comments
[ 3.1 ms ] story [ 316 ms ] threadThe same is happening with SaaS. SaaS eliminates the need for in-house IT! Except it doesn't. It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl.
Not following fads and buzz is a huge competitive advantage in this industry. Founders and chief engineers / CTOs / CIOs take note. Just make sure you can explain why you are not using (insert latest buzzword here).
The bottom line is that you should analyze the situation using your work load, your numbers, your culture, etc., and decide what works the best. Sometimes that's managed cloud. Sometimes it's unmanaged cloud. Sometimes it's bare metal. Sometimes it's on-prem. Your mileage will vary.
1) It takes amazing intestinal fortitude to fight back against the fad tide. And, when things go wrong, your decisions get the blame even if they aren't at fault.
2) As the article points out, cloud is FINE for startups and small companies. If my startup company reaches gigabucks in revenue and I now have to worry about $75 million in cloud spend, I've done my job. And then some.
3) Cloud is often about blame and liability transfer. I don't want the company website getting hacked to be my problem--I want it to be somebody else's problem. I'm willing to pay for that.
You pay a higher price for what you use than if you maintain those tools yourself... but you don't have to maintain those tools yourself. It's hardly a buzzword, it's almost defacto standard nowadays.
> Cloud became the fad, and the buzz was that cloud saves money, so everyone goes cloud
Cloud services _do_ save money for businesses of the appropriate size (meaning, those that can't or shouldn't be focusing on physical servers and networked hardware).
> The same is happening with SaaS. SaaS eliminates the need for in-house IT!
Again, SaaS _does_ eliminate the need for in-house IT _for certain classes of businesses_.
> It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl.
The cost of employing someone to babysit a SaaS product is much lower than the cost to employ someone to build and maintain an equivalent in-house SaaS product. If you're fine with vendor lock you're saving money.
> Not following fads and buzz is a huge competitive advantage in this industry.
Ignoring all nuance and claiming things that are popular are "just fads" is, to me, so much of a competitive disadvantage that it almost certainly outweights any perceived gain.
> So what can companies do to free themselves from this paradox? As mentioned, we’re not making a case for repatriation one way or the other; rather, we’re pointing out that infrastructure spend should be a first-class metric. What do we mean by this? That companies need to optimize early, often, and, sometimes, also outside the cloud. When you’re building a company at scale, there’s little room for religious dogma.
Yes, but the benefit of cloud is we only work to optimize those which gain market adoption. For every twilio there may be 100-1000 startups that did not make it, it's a good thing if those were constructed rapidly, tested for product market fit and then turned off without optimizations applied.
Also if cloud adoption actually accelerates building, then it may also aid its adopters if there is a race to product market fit.
Whenever a rewrite happens in software there are usually massive performance gains. The same thing happens with infrastructure.
That completely overlooks the actual benefit and the reasons driving cloud adoption, which he states early on and then fails to integrate here. The real cost/benefit analysis would take into account the amount of time, money, and opportunity cost saved by using a cloud in development, a critical time that determines whether there'll actually be a profitable company eventually. Optimizing the cloud costs is certainly important, but being able to spin up highly integrated systems on demand offsets a huge amount of time and capital during development.
A takeaway might be that, e.g., AWS, should offer even larger discounts for large scale operations in order to retain mature customers.
Seems similar to office space - a 5 person company will find the economics of coworking space compelling, a 500 person company will do better with a long-term lease with a commercial landlord, and a 50,000 person company might have their own property management team in-house. That doesn't create a paradox in the commercial real estate market, just different solutions for different needs.
Yes, it's not a paradox at all. Use this thing until it doesn't make sense, and then don't use it anymore (or be comfortable with the lack of optimization). This is not a paradox.
Why would this happen? Because what the company builds is a large and growing money-making contraption right inside the coworking space, also using parts of the building for critical functions, and the door is intentionally kept small. The only way to move that contraption is to dismantle it and reassamble in a much cheaper, purpose-built hangar, rebuilding some of the critical parts along the way. During all that time, the contraption would stop making money.
That's the beauty of AWS business model: it's a no-brainer for a startup to use it, but the startup grows, more financially efficient infrastructure options become unattainable because of the very high cost of the move. This is the best-executed vendor lock-in I know.
There are off the shelf things for, say, S3 and ALB which are entirely workable, but once you start getting into more complicated stuff (like S3's new consistency semantics, or SQS) then you're looking at a whole small company (at a minimum) worth of additional work. It's a non-trivial expansion, even for a large org with lots of money.
You can avoid using these sorts of services to maintain flexibility/independence, but you lose out on their unique benefits. This isn't an accident. It's not like you're going to be able to selfhost an SES clone and get the same kind of deliverability percentages as AWS netblocks, no matter how many engineers you throw at the problem.
Some of the goods you just can't get anywhere else, and they know it.
Having said that, and please keep in mind that I do not have an idea much about hardware so this is a guesswork; more of a question than a statement..., I do think that the Graviton price vs performance might not have a direct benefit outside of AWS. It's cool for AWS because they have less stuff to manage at that scale and it might be cheaper for them to buy and iterate because only they use it. It looks great to the clients because the perception is that the value is better. But AWS runs 80 AZs in 25 regions so it benefits them at that scale.
Realistically, even if one had couple of hundreds of racks, does it matter at that scale?
Aurora - tidb or vitess or cockroachdb or citus etc etc
Glacier - there is backblaze, also onprem vendors
Sqs - just use Nats
It's not limited to some. You absolutely need a whole new team (or teams) to handle your infrastructure, your high availability needs, and also your security.
The nifty serverless offerings and other features are just nice-to-haves in comparison to the core infrastructure work that cloud providers put into your system to keep it running.
Just because cloud providers like GCP and AWS and Azure and etc have everything put together to let you setup your whole infra by running a small script that does not mean nothing needs to be done in order for that to work.
SQS is a message queue service. It's a bit weird to claim that as "simply aren't directly available as selfhosted replacements", a message queue is pretty basic stuff.
I'm super into self-hosted shared-nothing queue services, but what AWS is doing with SQS is anything but basic stuff.
Small businesses don't care because they are too busy surviving.
Medium businesses don't care because why break what works?
Large businesses don't care because their managerial careers promote short-term priorities and marquee "transformation" projects which are buzzword-driven, and cloud is the current one.
Life goes on. It happens all the time. The trick is to learn and be smarter next time.
My company recently shifted from on prem to cloud. And we aren’t primarily a software company (although, much like most companies, we are increasingly turning into one).
We outsourced our data center needs, and I don’t remember us ever facing these middle of the night massive breakdowns (we had load balancing between a couple of data centers) and the number of ops issues hasn’t reduced a bit since the switchover.
I do think it’s been easier to improve performance for non US customers, because spinning up servers in data centers across the world seems a little easier around the world, but the millions saved by self hosting would have easily paid for someone to spin us up a Singapore and UK colo servers.
But you see part of the magic trick was Amazon spent years having companies like Garnter proclaim that cloud was cheaper, and nobody could possibly do on-premises IT cheaper because of economies of scale. As a result you've got CIOs everywhere trying to make a name for themselves by driving their company to the cloud to show immense cost savings. They don't have time to be bothered by the actual financial models or real costs. By the time finance realizes what happened they'll be on to the next gig.
It was honestly brilliant, the number of "cloud first" strategies that originated in board rooms filled with people that don't know the first thing about IT is kind of disgusting.
https://blog.min.io/repatriation_wave/
[0] https://techcrunch.com/2017/09/15/why-dropbox-decided-to-dro...
tl;dr: to save money (and to fix issues and irritations with AWS)
Do you not use IAM? Cloud specific infrastructure as code (again TF has cloud specific provisioners), permissions, compliance, training.
You can very much do that while staying in the cloud. It's all about engineering for flexibility.
Even small companies have multiple engineering teams, and each of those teams has particular needs, and each of the engineers, analysts or general users may rely on some service AWS, GCP or some other cloud provider has. Any single person can now break this ideal flexibility requirement if it makes their life easier.
> It's already demonstrated that the big cloud providers will cut you off with little warning if you run afoul of their political sensibilities. So the only way to use those services is in the most generic way you can, so that migrations don't cost more because of it.
You are stating that you moved from one cloud to another. The question is how you manage to build a company of any size that can just flip a switch and move to a new service provider without needing to transition teams to using new services and that costing money. If we're talking about moving a simple server from AWS to GCP, sure that's no problem. But if you or anyone at your company is actually using a cloud providers services for more than hosting a server, there is no possible way you can just flip a switch and migrate over.
You’re not getting retail price.
So you’re still spending more trying to maintain “cloud agnosticism” by hosting everything on VMs instead of using managed services.
Like Jobs said, Dropbox is a feature not a product.
For the same price you pay for Dropbox and 2TB, you can get the full Office 365 suite plus 6TB of storage or the entire GSuite.
And do you have multiple redundant DCs? Are all of your database writes duplicated 6x across three DCs like Aurora?
That’s just silly - dropbox is profitable if you exclude stock comp unlike box inc (who afaik runs in cloud) and also has higher margins (big shocker here). From techincal side i hear it worked out really well for them too.
> Like Jobs said, Dropbox is a feature not a product.
What does that have to do with anything? Not helping your argument at all
> For the same price you pay for Dropbox and 2TB, you can get the full Office 365 suite plus 6TB of storage or the entire GSuite.
Should be painfully obvious to anyone arguing in good faith why that is. Also do you by any chance think google runs gsuite on aws or gcp?
> And do you have multiple redundant DCs? Are all of your database writes duplicated 6x across three DCs like Aurora?
Yes, you run mutiple redunant colos/dcs and peer them. You’re making it sound like rocket science but it really isn’t. If Internet Archive can do it with their resources (not a dig towards their talent at all) so can unicorns flush with vc cash who supposedly hire “the best”
How many people who are comparing their costs to using a cloud provider are ignoring redundancy?
And I’m sure your average startup has the capability to maintain their own redundant data center. Besides, does it add business value? Would they really be better off hiring multiple infrastructure people or just paying a cloud provider.
It doesn’t matter why it’s the case. People keep bringing up Dropbox as a shining example of moving away from a cloud provider - and still not being profitable. But ignore a little company - Netflix - that did just the opposite.
Some commerce did need to scale fast and easily, and AWS provided that with great strength. But I will add that the management and executive branch were constantly and relentlessly seeking to undermine the bargaining power, day-to-day operations authority, and pay of skilled engineers who could do that kind of setup. Not that it was easy or risk-free .. I did see some powerfully strained eyes and nerves on those individuals who built and ran those kind of servers at various times. But do not leave out the manipulative and gaming motives of executive management to dumb-down admin requirements, seize authority via legal means, and seek the lowest wages across borders while doing so ..
Another good one is microsoft initially tying excel, word, access, etc to windows. And then IIS, SQL Server, etc to NT. An entire generation of businesses and managers got tied to windows this way.
In a way facebook and social media is a form of "vendor lock-in". The network/community itself and your "investment/time" in facebook makes it costly for many to move to another "facebook" or give up facebook altogether. The more time you spend on facebook and the more people you bring onto facebook, the more "locked-in" you are. It's quite ingenious and I suspect facebook understood this very early on.
At which point, as you say, the cost and complexity of a move is huge so you are stuck with a recurring bill.
Another interesting thing is that you pay for 100% of a VM virtual CPU even if it is only utilised 40%, but the cloud is selling your other 60% capacity to another user.
They have no incentive to charge you “based on CPU usage” for VM’s that are always on, even though it is possible. I think this would be a better approach than Lambda - to dynamically add more cores to your OS as you need them.
AWS don't overprovision CPUs outside of the t instance range, in which you get vCPU credits, not full vCPUs. If you rent an apartment but don't use it, is it the landlord's fault you don't? The landlord can't rent the same appartement to another client.
Especially on AWS/GCP/Azure, where there are many alternatives ( Lambda, Google Cloud Run, Google App Engine, etc.) where you pay per compute used.
One hardware core can be running 5 VMs at 20% capacity, and each VM owner will pay for 100% of their CPU capacity right? So in this case the landlord is renting your unused apartment to other customers then evicting them faster than you can open the door.
My point is cloud is benefiting from that ability and not passing it on the customers, which they could.
Its fully wrong for AWS, where a vCPU you rent == physical core/thread.
Are you certain of this statement?
Consolidate them, and a knife fight happens between middle management machiavellis and the one that wins is not necessarily the best on price or features. It's the one who is best at winning reporting hierarchy wars.
Instead they should pit these groups against each other as internal options for hosting to various services that provide what is needed for growth, adaptation, or cost.
AWS should simply be one of those options. Really it can be the prototyping phase, and then budget squeezing will move the app to other (hopefully cheaper) inhouse options.
What is frustrating from my years in the big corps is that AWS was cheaper than the internal "clouds" or other relationships.
I've had this discussion many times where I work, and we've refrained from using cloud services since they are extraordinary expensive compared to what we can do internally.
Yet, people seem to be convinced that prices are ok, and that going outside will let them reduce their infrastructure footprint.
For some reason I don't quite understand, people tend to be attracted by the cloud, even when it makes no sense economically whatsoever, and talking sense into them is surprisingly difficult.
So yes, it seems obvious that people will choose the most cost effective solution. However, it seems like they don't!
In my experience when a large corporation moves to the cloud it has less to do with pricing and more to do flexibility. It's easier to get a budget from IT and do whatever you want in cloud, then to have to wait weeks/months putting in a PO for hardware, getting access to those machines and installing what you want.
Devs look at Digital Ocean or Hetzner VM costing $10 for a ton of RAM, storage and bandwidth when the same thing internally in a bank or other big enterprise can cost $100-150. AND be delivered in 3 months, if it's ever delivered.
That's where you get it all wrong. It is not about the price, at all. It is all about avoiding large upfront capex with a small periodical opex, and in the process have virtually boundless growth potential.
Think about it: when you happen to need a bit more computational resources to run an app, is it easier to convince your boss the need to pay, say, 100€/month for extra VMs, or is it easier to convince your boss to shelve $2k to buy an extra rack? And how many times are you willing to have that same conversation with your boss whenever you run out of computational resources?
100% agree on shifting capex to opex though, capital works teams are super in favour of that, oddly enough.
No, not really. You can hardly find old used servers for less than 1000$, and you need to use beyond a a1.metal instance with 16vCPU and 32GB of RAM to even come close to that cost
> Cloud stuff only really makes sense when your resource requirements as super spiky.
No, not really. Cloud stuff makes all the sense in the world if all you need is a VM connected to the internet working 24h/day without problems. You can get that for less than 5$ a month, which would require 3 or 4 years to pay back if instead you waste your money on hardware.
> You can hardly find old used servers for less than 1000$
I picked up a second hand Dell PowerEdge 610 with 96GB of RAM for $400 AU, it's overkill by a large factor for what I need but damn. Either I was super lucky or servers aren't (weren't?) expensive.
> You can get that for less than 5$ a month pocket change and host it
Until you actually use it and then it ramps up rapidly to the point where you'd be far better off just buying a server.
And good luck selling to enterprises when the vulnerability scanner lights up like a Christmas tree with critical and un-patchable flaws on your switches/routers/UPS/PDUs/storage arrays/servers/door locks/cooling system/fire suppression/cameras. Double for the DR site. Because you need all that stuff, and you have to maintain it all 24x7 which means lots of salaries.
And if you're wrong about those estimates, the costs are real. Overinvested in hardware upfront and your project will never break even. Underinvest, and when the marketing campaign kicks in and you get flattened on cybermonday, there is no limit to how much revenue you might miss out on.
I'm willing to pay more money to reduce the downside of my being wrong. Cloud services offer so much flexibility that I don't even have to make guesses about some things beyond my information horizon.
On prem might be cheaper but it is far from an equivalent service.
All this happens willingly unless growth happened with your eyes closed, or really had no choice keeping up with it. Sure, use the cloud, but don't use every cloud-proprietary convenience. Use opensource software in a near-standard way. Even if using the cloud-vendor's offering, refrain from using proprietary extensions. If your plan is to grow and get out before things stabilize, then that's on who's come onboard, not resisted the conveniences, and remaining.
Moving office is relatively simple, you call a moving company to lift all your chairs, print new business cards, hire a bigger maintenance crew and it's done within a week.
Moving cloud could in worst case mean a 6 month complete rewrite of your core product depending on how much dependent you are to CosmosDB, Lambda or other shiny AWS stuff.
You could say buying a machine you previously rented is the same thing but when looking at code migrations, rewrites and launching new infrastructure cost time from R&D, not just money.
While repatriation can make sense at a larger scale company, startups and SMBs can yield the same benefits discussed in this post by simply tracking and optimizing cloud spend.
We try to make this as easy as possible for people with https://www.vantage.sh/ - where we're already helping thousands of individuals, startups, SMBs and enterprises as it relates to AWS.
The work was very interesting, because a lot of it was actually building a private cloud for internal customers & the work primarily centered around a virtualized data-center aimed at boom-bust cycles of games (15+ million users for 6 weeks, drop to 2 million for a month and down to a million in another week).
The issue is that the infrastructure cost is somewhat constant when dealing with that sort of fluctuations in revenues, so the cost to revenue ratio was unpredictable (while the cost was).
So what happened in the end looked a lot like a fire-sale of hardware when the cost was unbearable, while if it was an end-user cloud, that low-demand phase would be able to cut losses as a spot instance or something.
Anyway, a few years after I left, back to EC2 it is[1].
[1] - https://aws.amazon.com/solutions/case-studies/zynga/
The salaries and hardware cost was paid for even if the games had a bust.
The period where it worked well, the games had boom-bust in somewhat controlled fashion where farmville -> cityville -> frontierville -> fishville etc, the traffic would move around rather than die down entirely.
The world turned mobile-heavy and that whole pipeline fell apart while they were restructuring into mobile games (words with friends etc), when the hardware had to be sold or the payments would start to hurt.
It took me about a day to scope out a BOM that would well surpass that (maybe a year's expected growth) of whitebox server gear, and get some quotes from nearby co-los.
IIRC hardware capex was about £15,000, and monthly rack + network something around £3,000. We relocated within a few weeks.
One small bonus was predictable & consistent performance -- back then, the EU-west EC2 offerings were extremely sensitive to noisy neighbours, and our benchmarks never gave anywhere near the same results twice.
I think for genuinely elastic loads, or if you're really addicted to some vendor-only services, it certainly makes sense. I suspect most customers are overly optimistic about how elastic their requirements are, and their stack can be.
There is a learned helplessness when it comes to companies running their own data centers that has become widespread over the last decade. What used to be a fairly mechanical process has almost been mythologized as some kind of arcane art beyond the technical ability of any company that isn't Google or Amazon. Designing data center builds isn't difficult, it is a pretty straightforward albeit detail-oriented blue-collar engineering skill, but it seems few people learn it anymore.
[0] https://news.ycombinator.com/item?id=27294471
Your pitch emphasizes features that make servers easier to manage, presumably lowering (but not eliminating) the cost of personnel, so you're obviously aware of this, but I'm not seeing any "net of additional, fully-loaded personnel costs" in their analysis.
It should still be cheaper above a certain scale, but the breakeven where it will make sense to "repatriate infrastructure" is much higher if you include hiring people, and will go even higher if the cloud providers run the numbers and match most of the savings with scale-based price breaks.
My team had 2 guys doing SiteOps. They would travel to the various DCs in the Bay Area and Virginia and do all the maintenance, new installs, etc. And sometimes we'd lean on the colocation remote hands to do a few things.
We had about 5 network engineers, that also handled the corporate network. (12 offices and a network backbone that connected east and west coast DCs, offices, etc).
And maybe 2 SWEs who handled things like our host OS install system, etc. Basically the next layer above the hardware.
So 9 people that were required to run all of that stuff. But really, NetEng ended up spending like 70% of their time on corporate network things because we'd add offices faster than datacenters.
So if we focused on production only, we really needed about 6 or 7 people total.
I did the math a few times (every single year) and compared our costs, including people, to the costs of moving to AWS 3 year reserve instances.
Doing it ourselves was always half the price.
Of course the difference here was we built on-prem from the start. So there was no repatriation that had to happen.
Since then, I've been responsible for large cloud infra on all 3 major providers and learned a lot about what kind of discounts you can get when you're in the double digit millions in annual spend.
I still think, at a scale of single digit thousands of servers, you'd be cheaper on-prem, fully loaded. But admittedly, I haven't run the numbers since 2017.
And those companies are in situations where either people have no idea that their teams are performing badly, or lack the political will to do something about it. Adding the services in AWS is just easier, and might be worth every penny.
When you go to a high performing firm and explain the procedures and decisions of a low performing one, people think you must be lying, but it's true: Broken IT departments are everywhere in enterprise companies, and paying double of what a good team would take to do the right thing is a bargain, given that they can't even get a whiff at a good team.
If this is the case then the CTO knows that something is very wrong.
I don't think your scenario of companies not knowing whether their teams are incompetent makes sense. You can apply this logic applies to all functions.
But most firms don’t know how to employ a you to show them this way.
On the other hand, at merely single-digit servers, I'm guessing the fixed cost of personnel referenced in the GP comment would dominate.
Kubernetes only makes sense from Google's perspective to sell you the managed version "from the creators of Kubernetes!" once you get tired of trying to wrestle it into submission.
K8s isn't that complex from a cluster operators perspective if you've managed bare metal Linux systems before. Set up some PXE + auto provisioning and you can make your node enrollment process:
1. Buy hardware from an OEM.
2. OEM ships you a piece of hardware.
3. Remote hands plug in hardware.
4. Machine PXEs, boots, and is turned into a worker on your cluster.
Canonical (MaaS), VMware (vSphere), and Rancher Labs (Rancher) all have products aimed at running Kube on metal in your own datacenter.
So, essentially, if you start your startup off using Kube in a managed cluster you can, at some point in time later, directly move your code off the cloud and onto your own machines for cost saving. You just need to figure out how to run your DBs and LBs but that's a solved problem as mentioned above.
k8s lets us move fast - we can easily deploy new services in no time. We can also run the entire stack on our desktops in k8s, or have our laptops be a node in a mostly cloud k8s dev deployment and debug services locally.
Everybody loves to cite DropBox here. The greater arc of DropBox is their product stalled, they lost the enterprise of their market to Box, and they found themselves a commodity in a commodity market. Heck, maybe they'll go back to Amazon if they keep doing things like this: https://aws.amazon.com/solutions/case-studies/dropbox-s3/
But that's not all. It takes a top-level directive to repatriate an entire SaaS. That's at the cost of other top-level projects. It's wild to me that any company that has significant fuel left in the tank would buy back single-digit COGS percentages instead of investing in product that could add double-digit growth for several years at scale.
Yeah but why do they have 34 PB of analytics data?
The most powerful tools we have are aggregated reports and data retention periods, the same since the 1970s.
Running 10 copies of the same report (but each one with slightly different bugs) across 34 PB of data for 10 different departments is absurd, yet here we are.
Specifically for Dropbox, they're not a retailer like Safeway with 10,000 products and 100's of locations to crunch product and shipping data for - Dropbox could use a single server for their entire digital products data warehouse.
(I operated a data warehouse myself (literally) at Yahoo - 10 servers generating canned web reports for 500 managers. I heard they replaced it with a 1,000 node Hadoop cluster later.)
So not sure if it's so much that the product stalled, but that leadership actively didn't want to move in that direction for too long, then lost the advantage.
Instead they spent a bunch of energy on Carousel and Paper. (I don't have any idea how much of their eng team was working on those things)
Dropbox is worth about 3x Box as of today
Most businesses who are operating in the cloud are not so directly purchasing a cloud resource, layering a little value on top, and reselling it; most businesses make their money selling something else, and using cloud compute/storage/bandwidth/infrastructure is not a transactional cost.
For every gigabyte of storage space dropbox sells, they have to buy a gigabyte of storage from someone. And if you're looking ahead to the future of selling lots more gigabytes, you're quite motivated to find a cheaper way to buy them.
So I am never convinced by the argument that because Dropbox left the cloud, that means other businesses should.
Its good that we have people focused on lowering cost and optimisation, thats why people in India can afford a smartphone.
On the other hand, hisotry is filled with features and products that never found their niche, from random dropbox features to thousands of products google created and killed
This article only looks at "seen" costs, and assumes that there are no "unseen" costs to running on-prem. Many companies do not have the operational maturity to run on-prem well. The result: high cost of operations, low availability, and large increase in time-to-value.
Second unseen cost: everybody becomes their own SI. So far nobody really sells the "whole stack" for running on prem. I mean hardware, network, virtualization, application, traffic mgmt, etc. I have to buy stuff from two dozen different vendors and cobble it together into a high-labor, rickety Jenga tower of stuff.
If a company was going to run their own data centers, they would presumably hire someone that knows what they are doing to lead the effort instead of trying to do it by trial and error.
Yes, you have to plan, yes you have long lead times to get new gear, DC space, etc. But once it's running, failure rates are pretty low. At least in the single digit thousands and servers.
I've had to deal with much more frequent and odd types of failures on cloud infra than with on-prem.
Supply chain management and coordination is, IMHO, the most difficult part of it and is often overlooked in these discussions. Herding vendors to deliver hardware on your timeline from overseas supply chains doesn't always turn out the way you want, even with the best planning.
Isn't that the target audience for this sort of change?
I mean, do you see teams of networking , siteops, high availability, and security engineers hanging around doing nothing and just waiting for a company to decide to go in-house?
No, because those teams do not exist in free-range. That's something a company needs to build and train and experiment from the start until they are able to learn all the lessons.
My company (which is in fact part of the charts in the article) had every single metric across the board spike 15-20x basically overnight when the pandemic started last year in March. Our entire infrastructure burden was clicking a few buttons on the AWS console and making sure everything was provisioning and scaling as needed. If we had to send out people to buy hard drives and server racks at that time, there is no chance we would have been able to meet the extra demand.
Plus, if you give me a few dozen capable engineers today I'm not going to waste their efforts on rebuilding AWS to get a best-case few percentage point return on our cloud spend. I'll launch a new product for our customers instead.
I’ve also worked for a place so large in AWS we hit walls where we hit Amazon’s literal physical limit and had wait for them to go out and buy the hardware for us to provision.
For Dropbox specifically I don’t see them being any form of special case, if it saves money and they obviously had operational experience so it makes sense
Public clouds, like AWS, have cut their storage costs by more than 50% since Dropbox built their own infra in 2015.
The last time I ran physical infra, we had sever racks that I'd originally bought, running in production 5+ years later.
Once you're past the depreciation schedule, they are basically free except for power and space. Yeah, at some point and scale, it can make sense to get new gear that is more power and space efficient to pack more into the same power and cooling budget.
AWS still lets me launch c1-c6 instance types. So they still have the older generations sticking around. Yeah, the newer ones are usually more cost effective, but you do have to do work to migrate to them.
Given that interactions to the cloud are through a relatively small set of known APIs, I'm surprised that there isn't there a service which replicates the APIs but with the endpoints being on "repatriated" hardware.
I’m looking for a mechanism which apes existing cloud APIs directly.
But there's also: https://www.eucalyptus.cloud/
Designing and implementing a highly available and resilient storage solution on-prem is much more harder task that will take 99% of time & budget to do properly compared to a compatibility layer.
If you self-manage, your capital investment is initially higher, and lower over time. At the same time, the effort it takes to reach the same results is always higher, and the quality of the end product may be lower, depending on how much service quality affects your product.
If you pay for managed services, your initial investment is lower, and higher over time. But at the same time, you require less effort, and you get higher quality outcomes.
This is obvious to anyone who has worked in the industry and done both. First, host your own service: JFrog Artifactory, Atlassian Confluence, GitLab, whatever. Now rapidly increase the demands on this service. As demand rises, quality will decrease, because it takes a lot of time, effort and expertise to build a very reliable hosted service. Now switch to a managed cloud instance. Suddenly, the service's average quality increases. Performance is steady regardless of increase in use. There are virtually no interruptions to your product or development.
The impact of a service's quality and reliability has ripple effects. If poor service quality slows down development, that means development quality will go down as people cut corners to try and meet deadlines. If the service is used for production, it means your product's quality will suffer, and that effects your bottom line. So a huge amount of the actual cost is not just paying for a service, but also how much business value is generated or lost due to service quality.
There is simply no way to replicate a managed service without becoming a managed service provider yourself. You have to become a whole new business within a business. It's like a yogurt company also becoming a dairy farm. Running a farm is not easy, and you will screw it up for several years. Seems obvious for farming, but for some reason people always underestimate this when it comes to technology.
On paper, the Cloud's value proposition is scalability. But in practice, the true value is actually as a force-multiplier for your product's quality, reliability, and time to market. (Time to market not just being "I launched my startup" but also "I released this new feature before my competitor")
Avoid doing undifferentiated work. Outsource it when you can. Focus on the core product. I think this is especially true as you're scaling a company in the current SWE labor environment. If your business is growing quickly, you probably have money, and can get more funding when you need it. But hiring more SWEs is always a challenge. That is your scarce resource. Don't waste it on undifferentiated work.
All that said, at larger scale and more mature businesses. I think the core thesis of the article makes sense. You may be in the slower growth part of the S curve, but you can still increase your company value and profitability substantially by increasing margins. One way do do this is to invest in cheaper infrastructure.
Then you're taking on some undifferentiated work, but in service of better margins.
I've worked for a handful of very large businesses (10s of thousands of employees, profits in the billions) running infrastructure and services. With one exception, they all sucked at managing their own hosting. The only one that did well literally poached all the core employees of a web hosting company, and then listened to them.
Except for that one company, none of the others was willing to build their own internal managed hosting company. They all didn't staff right, didn't fund right, didn't do support right, and their org structure and financing model had no capacity for a single department to straddle every BU. (Well, ok, "IT" often did, but "IT" wasn't directly managing and running the underpinnings of every product in the company)
In fact, some of the largest, most well known companies in the US actually moved to the cloud specifically because they sucked so bad at running their own shop that they needed to stem the bleeding by just buying Cloud services that worked. And that's after they hired consulting/management companies to try to jointly manage their self-built setups.
The article makes sense. The idea that larger businesses can figure out how to do it right at scale, sounds logical. But my whole career is one long proof that those theories don't match reality. In fact, I would go so far to say that the larger the organization is, the harder it is to self-manage. The only general category of company that I think is OK to self-manage hosting is one where either almost all of the profit is coming from keeping hosting costs down, or the business value is not impacted by hosting quality at all. I think that's a very small proportion of tech businesses today.
> tie the pain directly to the folks who can fix the problem
That's why we're building https://github.com/infracost/infracost for engineering teams (free open source)