People seem to be posting hard-paywalled links more and more around here. They were supposed to be banned, but I guess the rules are selectively enforced by @dang here nowadays. The link to the article should be changed to this Outline link.
And it's still cheaper than public cloud. Look at cloud provider margins, that profit is your opportunity at this scale (and BoA, JPM, etc are big enough to run their own technology teams, and do, having worked with them). Fat public cloud margins come from somewhere.
This isn't a religious argument. Some workloads are fine on prem (known load profiles, shed infrequent traffic spikes instead of bursting), some need cloud (Netflix.com, remember all of their video content streams from their own CDN hardware), it's ensuring you're making the decision in an objective way. Sort of, but not quite, similar to making software budget and timeline estimates.
If software is eating the world, and every company is turning into a software company, why would you be surprised that orgs above a certain size wouldn't also turn into an infrastructure company? You don't have to compete against Amazon, you just need to satisfy your engineering needs for your vertical and the software that drives it. Amazon needs to be everything to everyone.
Because of the capital and talent required to build a competitive offering. Even large companies like IBM and Oracle have failed to keep up.
The big 3 clouds have built a nice moat and can charge whatever they want. They still compete on prices but it's clear they're colluding on network fees at least.
It's not just capital or talent. I have seen first hand non-tech executive decision makers pick a specific cloud provider because of the old adage "No one got fired for buying IBM". That momentum is incredibly hard to compete against. Google Cloud is fairly solid from a technology perspective, not to mention Google's deep pockets, and still has an uphill climb against MS and AWS.
Not directly a response to you, but GCP is actually phenomenal in some areas over AWS: BigQuery, Data Studio (I know it's not really "GCP" but vs AWS' offering), and Pubsub come to mind.
Oddly enough, they falter in other areas: no equivalent to Kinesis Firehose, and true to your point, I was told by customer service just to build my own DataFlow job to take a Pubsub topic and make files.
It's like some of their product managers have a very customer-first mindset (BigQuery, Pubsub) and some could not care less.
There's not much competition because of enormous lockin effects. Once a cloud provider is chosen, you're pretty wedded to it.
Inside your dev team, every institutional bit of knowledge about how AWS works, how to configure it, how to work around their mostly bad docs, and how to navigate their hot garbage UI.
Every line of code that touches an AWS api.
If you need data centers in certain regions for eg localization laws, AWS is afaik the only choice.
It is a bad title. The inclusion of "ignoring Amazon" implies that the difference between the using AWS and not using AWS is $2B. Otherwise, it makes as much sense to title it "Bank of America saved $2B per year ignoring shark attacks, building own cloud".
The savings implies they saved $2B over AWS. I can't read the article since I don't have a BI account, but unless they saved $2B over AWS (rather than $2B over their previous infrastructure) it's definitely misleading.
If you know your traffic well, private cloud saves a lot of money. Few years back i bet hybrid clouds will take over and for some reason it did not happen. I still believe hybrid cloud is the solution for mid size and up companies. You definitely need cloud provides for handling traffic spikes.
I'm sure at least a few companies have gone bankrupt or people have gotten pink-slipped for using AWS and not paying attention to auto-scaling settings...
It's scarily easy to accrue huge bills if you're not watching things closely.
I am not at liberty to say the company name, but i do directly know of one person at a director level who was fired for using AWS.
They turned on encryption for an option and did not realize the huge price difference between the encrypted and unencrypted option meant $9000 per day in additional charges. After 1 month (30 days) this came to light, and he was fired.
Yes, this happens with AWS. Controlling costs is a very important part of using any cloud, let alone AWS where the charges are... not as transparent as one would wish. :)
EDIT: btw, he was not fired for using AWS, he was fired for making unnecessary costs.
Part of the organizational allure of migrating to the cloud is being able to rationalize headcount. Hybrid clouds, while bringing a lot of potential upside, are complex. While you can potentially reduce the scale of labor involved, you still have to retain all of the core skillsets you already have to maintain your on-prem locations. Then add headcount to optimize the cloud deployment, _plus_ the ever elusive skillset of the individuals that can marry the two in such a way where you show operational savings from the hybrid model.
Private cloud usually refers to that same abstraction layer. Hardware provisioning is abstracted away from use and available on-demand from a pool, and likewise deployment works automatically via the same or an equivalent API.
That's very different from the operation of a typical company-owned datacenter from a couple of decades ago.
I think in this context it relates to the overheads of provisioning new services.
Traditional virtualized hosting environments typically also come with varying levels of technical overheads when setting up new boxes.
What bofa have is a virtually self-service (some business approvals needed) user interface with the ability for devs to set up new virtual servers with little/no knowledge of the underlying hosting infrastructure.
This is completely false - I imagine they didn't take into account how much the migration and upkeep actually cost them. AWS's promise is not that its infrastructure itself is cheaper in terms of number of servers, or otherwise. Its cheaper in that it helps automate a lot of manual things away.
2 Billion per year buys a lot of R&D. So, while Amazon is great for small to mid sized companies, BoA is huge and would need massive discounts to consider using Amazon.
Amazon can invest heavily in new tech because it has millions of customers who get the benefit. BofA can only invest in new tech to the degree that it benefits one (admittedly large) customer. This makes the decision seem more suspect in the long-run.
To the degree that BofA has needs that are unique just to itself, and do not overlap other large business's use cases, this is true. But that seems like an awfully small degree.
A billion dollars sounds like a ton of money on HN, but that's a thousand person tech team for 3 or 4 years. It's enough to do some good tech, but it's very, very easy to spend that much engineering capacity.
This is 2 billion dollars per year. That’s 6-8,000 people working full time indefinitely. Further they don’t need to spend that on staff they can have a large dev team, better hardware, more bandwidth, a 100 person support team 24/7 365, and still save money.
If you manage things right its not that hard. Unfortunately there are not all that many good devops people. Expect to pay them as much as higher end devs. But overall you can save a ton of money hiring some of those and passing on the outrageous AWS markup on CPU and bandwidth.
I had experience in Quartz (BofA cloud), and deploying is 100 times easier than AWS, everything is automated. Imagine building a cloud service where you trust all your clients, and all must share the same information if they have the correct auth, what I trying to say is that their use-case actually made it simpler, that's where the savings in software had came from.
So we'll hear about a BoA hack in a few years where the attacker got into the cloud somehow and then had unlimited access to all the other servers? Great. Exactly what I want from a bank. /s
Or you can look at it as a single correct auth and encryption mechanism is shared company wide vs each individual teams intern inventing new was to base64 encode your password. Glass half full or empty
Well that's the same with any cloud provider, one compromised account with enough access could be catastrophic. In BofA there is an AWS IAM equivalent. Also the BofA cloud is not accesible from the Internet.
There's definitely financial savings to be had by doing it in-house (either literally or utilizing existing data centers as here). But what I've experienced is that companies are often attracted to Big Cloud™ not just for financial reasons but to flatten/simplify their corporate reporting structure. It is the ultimate delegation.
This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously).
These "soft" costs/savings are rarely discussed because they cannot be measured, unlike a balance sheet. But there's definitely a whole other dimension to infrastructure outsourcing that is worthy of note.
In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks.
Also reducing friction in getting products to market. A small team with their own BigCloud account is probably going to deliver a product to market much faster than one that needs to go through the friction of dealing with the various IT departments which invariably exist at large companies.
There is absolutely value to delivering a product to market a year or more sooner than otherwise.
The friction from various IT departments are often beneficial. I’m probably speaking from a point of bias as I’m one of those departments, but we often have to sanity check the implementation of things and review the security implications.
I’ve encountered quite a few devs whose only focus is on achieving the specified goal without considering the full security scope. I don’t blame them, that’s what they are there for. Stepping in for ensuring the customer security and regulation requirements is what I’m here for.
But at a large company, how likely is it that a team is going to have their own AWS account AND have free rein over it? You'll still have all of the friction for compliance and security. And turf battles.
It's like using package managers like Maven and npm. Sure, they make going to market much faster for small organizations. But at companies that have strict security procedures, downloading unvetted stuff from the internets and putting it into production does not fly. You'll have internal repos that work with your package manager tools, but nothing gets into those repos without at least a cursory review.
But at a large company, how likely is it that a team is going to have their own AWS account AND have free rein over it?
The two companies I've used AWS at both operated this way. In fact, each team has multiple product accounts (for dev, testing, production). So I don't think it's unusual at all.
This is a false dichotomy. There is the issue of restrictive IT departments, legal review, compliance review, extended QA, documentation, product literature, etc. all extending a product launch timeframe...
Separate from all of that there is a valid question whether the software stack at a big cloud provider gives your engineering department the ability to deliver features faster at scale and redundancy, and with lower ongoing maintenance overhead. I think this question is very much up for debate.
At a small scale there are absolutely cloud frameworks that let you deploy services without an IT department at large scale. But then there’s a realistic argument of whether the high premium for theoretical scale—when in actuality you’re serving dozens not millions of customers—is worth the high cost of capital.
E.g. several startups of mine have persistent on a couple colocated dedicated servers for over a decade at a cost of $300/mo. The equivalent deployment in the cloud is closer to $3,000/mo.
I know this because I got $300k of Azure credits to play around with and ended up not using most of them. The long-run cost was too high to move to Azure. But there were months when I “spent” $6k of credits on a dozen VMs and associated storage and network that had equivalent performance to my rented bare metal.
...but they are also acumulating know how which they might be able to sell in decades to come. Even more if the scenario of mainframes, IBM software, Oracle databases,... will happen and sooner or later I believe it will. Currently cloud providers are operating on absolute minimum, to get as much customers as possible into their trap. And for sure, it is highly beneficial for startups that cant afford to buy on premise hardware, but this wont last forever. History has shown that immediately when you are vendor locked, you will be charged for your dependance. Even more if people beeing able to build on premise infrastructure are becoming rare beasts. I would be glad to see I am wrong but history proves it is best teacher.
Operating at a minimum? How do you come to that conclusion? From a pure cost of infrastructure perspective (ignoring all the human elements), the cloud is always more expensive than any other option. It’s got some of the most ridiculous margins and price gouging of any business out there.
I think the opposite will happen. Cloud providers are becoming commoditized. Kubernetes, blah blah. Developers hate lock-in, and on a long time scale, developers drive tech decisions. Nowadays new software is cloud agnostic, and it will continue to be that way. This is commoditization, which generally drives down prices.
That said, buying commodities from a market with four participants is more like buying from a cartel than a marketplace. I definitely worry about the consolidation of the world’s IT spending going into the coffers of three or four massively profitable and morally corrupt corporations.
Great call out for Kubernetes. I’m a full believer that it will turn cloud providers into dumb aggregations of compute/network/storage (like simple utility companies) for the deployment of CRDs and Operators, which will obviate the need for AWS services like EBS, Route 53, RDS, and many many other offerings. This will lower the cost to entry for competitors and lower prices for everyone.
A good chunk of the internet is already running just because Bezos alows it that way. The prices will go up substantially the day Lauren Sanchez walks away with his billions.
> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks.
I think some companies, like BOA, look at this as the safer alternative.
We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world conflict. These global companies with outsourced IT will be in ruins because politics and digital DMZ's will keep them from acting cohesively.
For an organization like a bank that's worth over $400b that layer of staffing, equipment, oversight, and all the inefficiencies are a $400b insurance policy that you will be able to continue operating regardless of geographic or political events outside of your control.
Exactly. Jeff Bezos tweets something that gets Xi’s panties in a ruffle and AWS is blocked in China and Bank of America can’t do cross border payments... no way should such a big company have their eggs in such a basket.
It was about decision maker's personal (career) risk when making decisions. Keeping things in-house means more decisions are taken in-house which increases personal risk to the management for getting things wrong. In context this is much clearer:
> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously).
The post above doesn't discuss cloud Vs. non-cloud security/safety at all.
> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks.
I've been at multiple Fortune 50 companies at multiple levels. Many times these types of infrastructure building are just forms of empire building and an attempt to embed ones' services in the organization at a very deep level. You convert one set of costs (AWS/GCP/Azure bills) for another set of costs (Infrastructure + Payroll). You also convert one set of headaches (CSP overcharging/auditing) for another (prima donna internal infra engineers.)
What's worse, someone being a prima donna or the unchangeable facts of AWS operation policies? You will always encounter roadblocks and misfeatures, but can you get around them?
Don't get me wrong, most corporate IT is deeply incompetent at their core job. It's just that you can't fix the deep problems by outsourcing the computer part.
> What's worse, someone being a prima donna or the unchangeable facts of AWS operation policies?
There's this great thing about BOA-sized support contracts where you get to call up the senior product managers and demand that they build the features that you want. And if you're not happy about something, engineers will drop what they're working on to appease you.
I wonder how many customers were interested in setting up a Custom Keystore for KMS on CloudHSM?
Can confirm, I'd say 20-30% of all Azure features come directly from the product teams' Partner Advisory Council (read: top customers) meetings. Like "snap your fingers" implementation.
Hopefully you'd think they compared those costs and still found the in-house option would be cheaper in the long run. Who knows, though, or who knows how accurate their estimates were.
Once you are deeply embedded with the in-house option, you have no competition. Sometimes distinct layers bleed into each other and things get even more messy to untangle. Also, sometimes due to embarrassment alone, you continue with the in-house option.
With CSPs, at least you have four major players (AWS, GCP, Azure, Oracle Cloud, ?IBM?) competing prices downward and multiple theoretical options. Yes, you have your in-house CSP expert biasing things, but since most of the work is external, at least you dont have a heavy internal weight on a home-built choice.
IMHO, the better option would be a CSP-agnostic Kubernetes based layer. But IMHO we're not there yet.
BOA also didn't (doesn't) really trust the public cloud. During several all hands meetings in the org, the inevitable question of "when are we gonna start using the public cloud" came up and the leaders would always say "we aren't because it isn't secure enough" or "we'd rather maintain our current governance model over what our devs build".
To your point, we need only point to recent events to show the universal customer pain that can be caused by a bad code push at Azure or AWS.
A factor that is easy for developers to overlook is that the legal system still cares who owns the hardware. So if all your stuff is at Amazon, a warrant can be served to Amazon and they will let the cops into your stuff. No contract can keep that from happening.
If a company (or person) really wants complete visibility and control into their relationship with law enforcement, they have to host their own hardware... ideally, their own data center.
I've always wondered about this. In IT now instead of having 5 system operators one has 10 "cloud consultants" and nobody knows what they are doing. I've been through a couple of "migrations to the cloud" now and I've never seen a reduction in IT staff or the cost of running IT operations.
About the wrong decisioning part, if you chose the wrong "cloud" for your business the cost to change can be enormous.
The primary motivation for management is from what I've seen so far the ability to point to someone else if shit hits the fan, not cost reduction and all that other stuff.
You start wondering if 90% of corporate world are there to do nothing if not erecting roadblocks for people who are actually doing something. But then you start wondering maybe it's a good thing that all of these people get paid and could spend their money so that majority of the country has purchasing power and not live in piss poor poverty.
> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions
This is a myth directly from cloud providers marketing campaigns: that using the cloud simplifies infrastructure to the point that you don't need as much and/or as qualified staff.
In reality it is quite the trap: without experienced system administrators AWS and the like are free to dig their hands deeper and deeper into the pockets of their unsuspecting customers.
Yes, this also seems weird to me. And I think it might be a cultural thing, I noticed that in Europe renting dedicated servers is far more popular than in the US.
IDK where you get the 'wages are a lot lower in Europe' thing. There are lower cost and higher cost places, but people are expensive and talent is mobile.
I don’t know, in the Bay Area 150k is entry level and big companies pay 250k or much higher to senior engineers. Whenever I hear about pay in Europe it’s a fraction of that. It may be worth it for a different lifestyle, I’m not making that argument. Regardless, from the company POV it must lead to different choices about buy vs. build.
Bay area or NYC sure, that's like being in London, but what about the Midwest, or any of the places where you can easily set up an IT shop without paying a premium for space?
- Get your own bedroom.
- 30 minute commute.
- Not everyone you meet will be in tech.
- You'll be rich compared to everyone else in town.
- Partner with local academic institutions to offer recognized research projects and training. 20% time or 3 month project stints etc. Adjunct professor ships for people with an existing research record.
- Throw in X free flights to east/west coast, home town etc.
there’s 30 minute commutes in san diego, la and sf if you live in the right areas. what about weather? cali has really nice weather. schools / doctoral candidates / etc are basically useless in the real world. if i’m making enough money the free flights are useless. nothing you said here makes me want to move to the middle of the US. exactly what i’m looking at is the lack of opportunity. say i decide to move for some job and hate it. now what? pick up the family and move again?
Yeah we literally turned our $7k a month AWS bill into $1k a month by changing to 3 dedicated servers (Fully managed!) and cloudflare. It's better in every conceivable way.
We were able to cram a large set of applications into a group of instances using CapRover.
Instead of 2-3 instances, and a whole bunch of RDS, we ended just recreating the databases there along with minio for storage (we write a small amount of data to S3 storage).
what instance types were you using? do the dedis have the same specs as the ec2 instances? what about contracts (do you have one on the dedis?) are you only using servers or other aws services?
Colocation is another in-between. Build your own server, but the racks / datacenter / power delivery / real estate / network is all rented from another group.
Colocation is probably ideal for anyone using custom hardware: like GPUs or FPGAs. If you're using "normal" CPUs, just buy dedicated instances instead.
Renting bare metal is exactly how the business worked prior to the rise of EC2. e.g Rackspace, Linode.
You can rent vs buy at any level : rent only machines, rent a 1/2 a rack, rent a full rack, rent a cage, rent 1/2 a dc, and so on.
Same with connectivity: you can plug your machines into a lan managed by the hosting provider, or run your own routers and peer with their in-house ISP, or you can buy peering directly in the building, or you can rent lightpath from a telco with pop in the building, or you can go rent a backhoe and start trenching across the parking lot...
Ya, I feel like this must be coming from the younger crowd who is just used to seeing all the cloud marketing comparing cloud computer to building your own data center.
Anyone who has been in the industry since pre-cloud explosion knows that most people just rented bare metal and at the more complicated end they might purchase their own hardware and colocate.
And for those that own their servers, many lease them and HP or Dell or whoever maintains them on site if anything goes wrong.
You can outsource the physical data center and connectivity without paying massive premium for a software stack which in many ways requires more expertise to operate at scale and redundancy than the in-house alternative, while costing an order of magnitude more money and locking you into a potential competitor’s ecosystem.
If you reduce companies to "developers hiring developers, led by a couple of business people," how resilient will that company be? I'd say it keeps them at a disadvantage for either elimination or acquisition, but dependent nonetheless.
Maybe cloud usage can be an indicator whether a company has plans to maintain independence into the future, despite the loyalty-engendering words of leadership.
Sure. You just need more people to keep an eye on what your consumption rates are. Or to handle access control systems or manage service accounts and client certificates, ensure firewalls, additional tooling around firewalls to ensure things aren’t getting exposed outside. Countless cost projections on top of a complicated billing system.
I mean. You save the headcount in one area and you consume it in another. And you pay overhead in outsourcing costs.
It’s likely worth it. But you need to determine that on a case by case basis.
My (large) company is too incompetent to do hosting properly so cloud was a blessing for us.
> Sure. You just need more people to keep an eye on what your consumption rates are. Or to handle access control systems or manage service accounts and client certificates, ensure firewalls, additional tooling around firewalls to ensure things aren’t getting exposed outside. Countless cost projections on top of a complicated billing system.
At a certain scale, you need all these things with internal clouds too. Resource provisioning, basic security, and managing "insider risk" don't just go away when you own the infrastructure.
How so? Presumably someone with an internal 'cloud' needs someone to be responsible for a large kube cluster, to handle a uniform way to do scheduling/resource management, etc. Having an engineer who knows how to do this in AWS doesn't seem more expensive than an engineer who can roll their own internal 'cloud' system...
> Countless cost projections on top of a complicated billing system
I think it's a loooot easier to design and build a cost-efficient multi-tiered high availability application than it is to make reasonable cost projections in whatever micro-currency every seconary and tertiary service uses.
And I'm ever surprised by how time-intensive it is.
Sure you do, you’re just blending them into the rate, at a significant margin.
Cloud is awesome for companies where the capital investment needed to deliver the SLA is too expensive.
If you’re big, and you run the numbers, lots of services are better in a datacenter that you manage. Many of the early advantages of cloud tools aren’t a differentiator today. If anything, the granular billing is a nightmare to manage in many large enterprises. There’s a reason AWS heavily targets .gov customers — the Feds probably waste billions on idle services.
I’ve found that the shitty to support or known workload services are best somebody else’s problem. Email is an example of both. Most other things are trading IT complexity with accounting complexity.
Counterpoint: Granular billing of real money can actually drive behavioral changes in ways that fake money and/or letting one department slide because you’re hoping it’s head will rubber stamp your promotion doesn’t.
> This is a myth directly from cloud providers marketing campaigns
I agree the cloud providers like this idea, but I'm not sure I agree it's a myth. Businesses need to choose what competencies to focus on. For nearly every other area, businesses rely on suppliers. Auto manufacturers use machines built by other businesses and parts built by other businesses. They also obviously do some amount of in-house part and tool design. It's a balance and I think that it's a lot more likely that most businesses will want a good partner to provide infrastructure (ultimately at margins less than AWS) instead of building it themselves.
> Auto manufacturers use machines built by other businesses and parts built by other businesses.
Yeah but they generally aren't renting those machines. Renting critical pieces of business infrastructure that aren't easily replaceable is opening yourself up to rent seeking on the part of your supplier. Remember the Oracle business model--lock people in and then keep raising the price just below what it would cost to rebuild from scratch.
Why is it better to be locked into your own company’s IT department rather than AWS? I would argue AWS gives a better ROI than 95% of corporate IT departments, and generally less hostile to boot.
No, it wouldn’t make sense for me to make the 95/5 distinction only to prescribe a 100/0 response. My post was a call for sober judgment, which is admittedly not very popular here.
Many people did not realize that IT infrastructure is a utility, it should not be owned all the time by most of the companies but purchased when needed. AWS couldn’t exist if it wasn’t better than on-prem IT. How better is defined is different case by case. Usually it includes TCO, ROI, turn around time, efficiency in terms of capacity management and fee more. These are the things many small companies do not have to deal with or easily solved. Yet there are many companies choosing AWS to solve these problems for them. In my experience AWS really start to make sense if you are spending more than 100.000 / month on infrastructure.
I don't think the renting example is that true. AFAIK auto manufacturers certainly do "rent" many of the factory machinery they use, and they certainly "rent" the maintenance contracts from the companies who service them (the big exception I'm aware of being Toyota, which makes their own factory robotics).
In other examples, many airlines do lease airplanes, and even among the airlines that do own the planes, they still usually lease the engines.
As the GP said, there's a balance to be struck. Delta doesn't have the expertise or the economies of scale necessary to build their own engines from scratch, but they do have the ability to lease those engines and then have their own teams maintain them. Similarly, most companies don't have the know-how or the EoS to run their own data center, so instead they buy cloud services and then have a smaller team to maintain it. It's a win-win. Some companies like BofA might be big enough where they can run their own cloud and it be cheaper than using a cloud provider, but that's definitely not a common thing.
Could you clarify "AFAIK auto manufacturers certainly do "rent" many of the factory machinery they use, and they certainly "rent" the maintenance contracts from the companies who service them (the big exception I'm aware of being Toyota, which makes their own factory robotics)."?
Rental only makes sense if there is a marketplace for used machines. Eg, planes and plane engines are components that aren't really tied to one place or company. (The "that aren't easily replaceable" from the parent post.)
But industrial robots for car manufacturing seems like rather specific hardware. If, say, the Ford plant in Claycomo, Missouri no longer needs a machine for making the F-150, who is going to rent it? How is rental a better cost-savings for Ford than buying the machine and later selling it?
Also, '"rent" the maintenance contracts' doesn't make sense to me. I sell maintenance contracts to my customers, I don't rent them, and I don't understand what "rent" could even mean in that context.
The difference between renting and purchasing gets murky (sure, not literally) when you are dealing with sensitive hardware that you rely on but cannot maintain yourself. Such systems usually require frequent maintenance by the supplier. Sure, you could end your maintenance contracts, but then the systems you have purchased will very rapidly become useless. The up-front cost of hardware is often very small compared to the ongoing cost of maintenance contracts, and so the relationship ends up looking a lot like rental.
The parent comment mentions that Delta rents their airplane engines. I imagine that they have the capability to do general maintenance on the engines, but that the engines need to be periodically sent back to the supplier (probably GE or Rolls-Royce) for a teardown, safety inspection, and rebuild. Without regular inspections an engine can't be flown, and is essentially worthless. Not much point in actually owning the engine if you can't use it.
I guess I have a naive view. I think of "buy" as something which is a capital expenditure, can be depreciated, etc. while "rent" are things which fall under operational expenditure.
With that in mind, I can see support contracts as an operational expenditure.
But if "over half of industrial robot purchases in North America have been made by automakers" then that's a lot of capital expenditure. I never got the feel that the companies which make modern industrial robots primarily rented them out (akin to IBM's renting out of tabulation machines 100 years ago). I really thought they mostly sold the robots.
They probably do sell them. I believe most industrial robotics systems are built custom, or at least heavily customized. But I have no idea (and I'm sure they closely guard) what percentage of their revenue comes from direct sales and what comes from support contracts. Somewhat related, I recall reading that one of the things that caused financial issues for GE Power was broken incentives around support contracts. To meet quarterly targets the sales team started offering to trade larger up-front prices for reduced costs on support contracts so as to frontload revenue. Too many customers saw the golden opportunity and GE ended up losing a ton of money.
Many of these industrial systems are built from reusable components. In my experience at a Tier 1 OEM for the US domestic auto market, a line could be torn down and re-purposed to another line by adding or removing physical components, modifying the physical layout where appropriate, and then adjusting the PLC programming.
An assembly line isn't really a "product," per se. It's more of amalgamation of several products and technologies, many legacy, that forms a cohesive but loosely-coupled whole. You may have several vendors' products / systems in a line. So what you end up with are ongoing contracts that provide for installation, programming, troubleshooting, maintenance, etc.
There's actually a whole specialized industry that does nothing but design assembly lines. It's pretty neat.
It's more common than you think. In industries that are locked into legacy apps, cloud costs are massive. Same for industries where storing massive amounts of data for decades is normal. Healthcare and banking spring to mind. I wouldnt be surprised if big pharma are private cloud too.
There are quite many different types of airlines. If you show up for some flight, it might actually be operated by personnel that are on the books of some other airline.
There's Boeing which also used to have an airline. It was made illegal IIRC.
And there are things in between these two extremes.
Some airlines have a heavy maintenance organization. Some have outsourced it. Different companies have different market segments and safety records etc.
I think a company that saves $2B (meaning they were able and willing to spend even more before they had this idea) is more than able to have multiple competencies. Acquisition and vertical integration is how many companies became massive.
Of course it's a question of scale, but a lot of arguments I've seen usually come down to CapEx vs OpEx accounting which is silly because a dollar is a dollar.
CapEx vs OpEx isn't about raw numbers, it's about who can make the spending decisions and how quickly they can be made. At most companies getting a couple additional servers is a lot easier when you're running on public cloud vs running on-prem.
You can use financing arrangements to avoid dropping the money all at once -- for example: A service provider offers you a monthly subscription or "lease-to-own deal" where you sign a contract to buy the software or hardware in exchange for monthly payments to be made over a period of time; that could be indefinite, or you become the owner of the hardware after the end of its useful life. Or just go to a bank, and they will be happy to create a loan and write a check out to your vendor for the lump sum, and so you are only responsible for making required monthly payments of the interest plus any principal, etc, required by the contract.
So no... not necessarily... CapEx is not necessarily paid out all at once. So the accounting ought to be the same whether its a CapEx purchase under a Subscription Agreement/Loan, or if its an "OpEx" payment for 1 month of services.
in the case of computer equipment that needs replacement very 4-5 yrs (or less depending on compute and other reqs) you’re perpetually in a lease and this starts to become opex
The general trend with cloud investment is that it makes utmost sense early in a startup or business expansion when time to market is the most important and uncertainty is high, but once the business stabalizes the assets move in house because of costs.
The cloud is powerful, but for long running businesses who can afford to invest in the competancy it's stupidly expensive. Hybrid cloud installations are ever more the norm, private clouds increasingly capable (PaaS, IaaS, SaaS), and toolsets like Kubernetes reshape how a lot of those legacy apps get looked at.
Why does this argument apply to AWS but not to desktop IT providers, paper office supplies, janitors, cafeteria services, and everything else that a business hires vendors for?
I don't necessarily think you're wrong, but I think you're missing the point.
Situation: some manager is tasked with updating some antiquated COBOL monstrosity. They're given twelve peanuts, some shoe string, bubble gum and a little bit of duct tape. The experienced staff have all been aged out, quit out of frustration or laid off and replaced with fresh college grads for half the price which was a great deal and significantly reduced costs for several consecutive quarters, earning a well deserved healthy bonus. Unfortunately reliability of the system has been suffering and the young engineering team unanimously agrees the old system should be replaced.
Option 1: do it in house with the young team of rockstars the manager has been bragging to management about. Unfortunately, the manager secretly has reservations and has doubts about the team's experience, let alone skills. It's too risky. Not too risky to the company who can easily absorb the loss and move on, but too risky for the middle manager who will look bad and possibly get fired.
Option 2: outsource the entire thing to IBM or Oracle. Tempting, but not in the budget. Also, significant parts of the old system are on IBM anyway and the cost of hiring IBM people to fly out has been increasingly expensive after the latest round off layoffs saw Scott put out to pasture. No, it's too risky. Not risky to the company who can have uptime guarantees in the contract, but risky to the manager who will look bad if he comes in over budget.
Option 3: The Cloud. It's the 'in' thing. FAANG are all all in on it, and if it works for them it could work for us. Much of the new team used AWS for their web dev class in college, so the skills are fresh. There's very little risk to it. The staffing requirements will be dramatically reduced so costs will drop. The manager will be able to deliver an awesome demo next quarter and the S3 instance they'll host it on will only cost one or two of their peanuts. The only risk is if the manager doesn't get promoted before the problems start, which shouldn't be a problem because AWS has such great onboarding programs. It is slightly more risky for the company though, who may find themselves no longer in control of critical infrastructure.
Risk for the company isn't the same as risk for a mid level decision maker. Small fuckups carry the same penalty as huge fuckups. (you get fired) From my fake story, doing it in house has a relatively high chance of being a minor fuckup, and AWS has a relatively low chance of being a major fuckup. The manager is going to choose the low probability risk, even though the low impact risk is the company's best interest.
And let's be honest... a remarkably large fraction of small IT shops are really bad at certain basic competencies, notably backup reliability and timely application of updates. AWS is pretty good at sysadmin and accounting 101, even if it stumbles at 201.
> Risk for the company isn't the same as risk for a mid level decision maker. Small fuckups carry the same penalty as huge fuckups. (you get fired) From my fake story, doing it in house has a relatively high chance of being a minor fuckup, and AWS has a relatively low chance of being a major fuckup. The manager is going to choose the low probability risk, even though the low impact risk is the company's best interest.
Insightful. Can you clarify what you mean by "low probability risk" and "low impact risk" ?
wholeheartedly agree. Also very few companies, need scaling like netflix. But hype of AWS makes inexperienced architect to choose stack without looking into pricing and complexity. Later projects are continued by throwing more resources on cloud.
Most medium size companies still would be using a data center provider today to handle the server/hardware maintenance and security. I think it comes down more to how likely is a workload going to change or interact with cloud services or api’s. If you have a CRM system used by a 15 person sales team growing one person a year, it probably is cheaper to self host in some ways. At the same time, if it goes down, someone internally needs to fix it. If all the sudden you want to run some data analytics project on the prospect and customer data, potentially the project is road blocked if more server space needs to be provisioned, etc.
If you already have IT staff in-house it ends up being a duplication of roles and an added layer of complexity. I get some servers and/or resources can benefit from autoscale. USGS gets slammed after a major earthquake so having autoscale capability is important. Or you want to co-locate some servers (payroll) for disaster recovery purposes. There are some legitimate uses cases, but often cloud just feels like renting servers that you could own and manage in-house.
I worked on the internal BOA cloud. My team wrote software to procure VM's via a website, versus the older (still current) way of doing it manually.
There are two things worth considering. The BOA cloud is slow. That is to say, it is hard to procure machines to get your projects going. There is a lot of control on who will be paying for them. There is also A LOT of staff to manage it. So those 2 Billion could easily be tweaked downward when you consider the staff and lost of agility.
On the flip side, AWS is hella expensive. To do the things that they do it would cost A LOT of money. Maybe even more than they think. I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills. So this number can also be tweaked up when you consider dozens of teams each having their way (at BOA scale) on the AWS console.
"I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills."
Ultimately,it doesn't matter, if the value it provides is greater than the AWS bill.
I am not sure I agree with that. Overpaying for something is still overpaying even if it provides value. Do you spend 100% of your salary on food? You will die if you don't eat. But somehow, the food required to keep you alive probably costs less than your salary.
not necessarily. for each of these “we build our own cloud” stories there’s quite a few more stories of large enterprises saving tons switching away from their own cloud. BOA themselves know their savings won’t last and they’ll be switching back. They’re already in negotiations
That's generally the economic case put out in cloud planning: a startup or expansion does great in the cloud with quick time to market, but once that business has matured and the risk is gone the assets move in-house because it's wildly cheaper.
And something that seemingly everyone forgets: it's not black and white... you can have some things in the cloud while you have some thing on-prem, leveraging scalable infrastructure and redundancy while also keeping your core assets fully under your control. It's pretty easy to make an internal S3 replacement, it's hard to make a better S3.
So they are not actually 'building' the infrastructure management layer from scratch.
Also for investment banks (due to exchange connectivity) , telecoms (due to peering points, physical tower locations) always going to have their 'Front-office' applications in data centers that offer lowest latency for those apps.
Midoffice applications (or BOSS as they are labeled in telecom) can live in data centers reasonably distant (in terms of ping time) from the front office.
So the infrastructure for those has a different cost profile.
Most of this applications touch trades or trade events that carry millions of dollars worth of contract value
(not Facebook's likes or favorites... )
There is a lot of logging, reconciliation, post executing enrichment that's probably going on.
Finally, from my 2nd hand understanding,
Investment banks can have from 3K to 10K applications in production. With, perhaps 20% of them having more than million lines of code.
Some of those apps are built in their own proprietray languages developed specifically to manage contracts and lifecycle of contracts (Banks had 'smart contracts' DSL and interpreters since 90s).
In my opinion, a company can save money on a cloud if it has these things:
- Access to cheap real estate
- Access to enough human capital, especially a team of 80+ to operate all aspects of a cloud: hardware, switches, openstack / kubernetes
- Requires scale of 20k+ servers = 320k+ VMs and a strong inclination to not host on AWS/Azure/Others.
- Lease to servers costing about $250 / server / mo. With AMD, you could go half of that, with VMs costing $10 a month. Add network costs of $15 / 2 ports / server / month.
I'm not really sure why this should be a surprise. Just as big companies use a mixture of owned and rented real estate for various reasons, the same is true of other large expenses. If you have a large, predictable, core workload it makes sense to bring it in house, and use the elastic (rented) resources for unpredictable stuff (e.g. new efforts).
It's not like Amazon doesn't know this too (e.g. they try to make the transition harder by offering lots of proprietary services that help you ramp up faster, but can't be used elsewhere)
There are plenty of reasons to choose a point in the space denominated by the axes make/adapt-existing X insource/outsource X proprietary/open . None are zero cost unless you really think your platform is good enough that enough others will write for it without you doing anything.
It's possible that you would chose to write your own because nothing in the market is even close to good. It's possible that you would open source it in the hopes that it becomes standard and that others will add new features that you will benefit from too (this is not a zero-cost option either). It's possible that you would even keep it proprietary because that's cheaper. But given the way Amazon operates it's pretty clear that they offer their own proprietary services to lower the barriers to adoption ("simply offer a better option") and raise the barriers to switching.
I don't see anything wrong with this strategy legally or morally, though I mostly avoid amazon proprietary tools because the risk of locking is one I'm not willing to embrace. There are of course situations where they aren't a problem (e.g. a tool with an anticipated finite lifetime: the cost of lock in in that case is essentially nil).
Agreed - past a certain (probably fairly massive) threshold of predictable long term load, it seems clear renting resources is going to be more expensive than the all-in cost of owning and operating your own.
Were the opposite true, you'd wonder about the economics of the cloud business model.
I think a more interesting way to look at this is, are the other advantages of cloud services (e.g. lots more flexibility and bundled proprietary technologies that you can't economically build yourself) actually worth the extra money?
Yes, by not using a cloud service you might save $2B a year, but does that cost you the opportunity to make even more than that, given you're probably moving slower or at least less efficiently than you otherwise could?
> but does that cost you the opportunity to make even more than that, given you're probably moving slower or at least less efficiently than you otherwise could?
The potential opportunity of fast-moving cloud features needs to be weighed against the opportunity costs of slow-moving cloud features. Where bespoke solutions can immediately provide tailored performance and maximize technical capabilities, a missing feature in any of your cloud providers services can be a showstopper or unmitigatable roadblock. And while lots of the technology is past the bounds of reasonable economical replacement, some of the technologies being shared through the cloud are nigh unfathomable to recreate.
Which is to say that the black ju-ju behind Windows update probably takes making a new MS to build up to present maturity, and unless you're a certified "big boy" letting small teams somewhere else fully dictate what you can and can't do at service boundaries probably impacts you in the long run.
Based on that, and IMO/IME: the answer isn't a binary choice but a constantly shifting point on a spectrum between the two, where on-premise/local-cloud and remote-cloud services are aware of one another and maximize capabilities while minimizing costs. Hybrid installations are just stronger, and are easier to reshape according to costs.
You’re also neglecting TTM as a consideration. As an example, I’m stuck having to wait 6 weeks to get new hardware provisioned into an available slot in one of my data centers to spin up a new Hadoop cluster, something my team could probably do in a couple of hours in the cloud.
That being said, we control the HW and SW stacks end to end so I don’t have to worry as much about the nightmare scenarios the NordVPN folks went public about today. Critical given we’re in fintech ...
Friend of mine at a payroll processor you've all heard of was excited to migrate to AWS because it took 4-6 weeks for IT to provision a new VM. On their app team's dedicated hardware!
So much of the Fortune 500 lust for public cloud seems to come down to working around inefficient IT procurement and provisioning processes. They haven't automated, they don't maintain enough excess capacity, they haven't managed vendor relationships to assure fast order turn-around, financial controls are too onerous, etc.
Everything's virtualized but otherwise they're still operating like it's 1995.
> Right now, the bank estimates its private cloud is 25% to 30% cheaper than public providers, though it also recognizes that probably won't last forever.
I am glad they shared this situation. We often here marketing telling us that public cloud is always better. It's nice to see different perspectives being shared and some details around them.
If you have $50million+ you can deploy on capex in one shot (economies of scale negotiation doesn't work well at lower numbers), and you can hire systems engineers (C/C++/Go/Rust 5+ years systems experience x 20 engineers), you can put together a single tenant compute cloud needed to run a typical Internet facing web/mobile application with all the bells and whistles (lots of app server clusters, db clusters, big data clusters, GPU clusters etc with HA, scalability, security, DR etc). At this point, capex/opex cost will be easily 30% lower than the best negotiated bulk price from public cloud.
It is worth doing this if your in-house engineering team is at least 100+ app service developers and have high feature churn rate.
But be warned, your systems engineering / shared technology team should be level-headed and mature and rest of your app service engineering should be good too to pull this off well. If not, you will be in serious developer productivity pain and there won't be quick and easy fixes once you have put down the capex.
Folks are generally misinterpreting the 'cost' of the cloud by mostly looking at data centre costs, margins etc.. The 'TCO'
or 'Total Cost of Ownership' of cloud services is generally much lower than otherwise.
The cloud is valuable because of the variety of features, immediately available, at very large scale, run by very credible people with $0 in capital expenditures.
In most cases, it's worth far more than the price, which is why it's so popular.
In very, very few cases is it 'cheaper' to run one's own cloud given the massive operational overhead of hiring people with very specialised skills, risks all over the board, the enormous planning, extensive investments.
Wallmart must have 1000's of trucks but they don't 'roll their own' delivery vehicles, because there's an inherent degree of specialisation in making vehicles.
So yes - in some specific cases it makes sense to 'host': say Dropbox, just wants massive cheap storage, or Google, has special needs & scale & ability, or some entity has very specific privacy requirements - and of course I wish there were more variety in providers, especially localised ... but the cloud is essentially now the default hosting option and that's not going away because it makes a lot of sense.
The surpluses generated by such services are vast - it's a big step forward for everyone. Companies using AWS are probably profiting more in surplus than the margins in AWS.
I continue to be amazed that any business of more than a few hundred employees would ever consider using off-prem hosting. Cloud services are there to bootstrap a business, not maintain it.
Well the issue is that you bootstrap on AWS and then have a hard time going on premise without a major rewrite or initiative. That and you want to keep adding features and driving the product forward... it is easy to see how this happens. Plus you can have 200 people focused on things other than running ESX boxes or docker or whatever you need.
Using cloud services saves a business from having to have a whole vertical stack of expertise in house. And for banks, these infrastructure engineers have to be available 24*7.
For many companies, hiring and talent retention is a problem. There are some risks that are not as superficial as dollar signs on the balance sheet.
Obviously small businesses don't have the option of building their own cloud, but there is a middle ground: use public clouds, but avoid coupling yourself to them. Most cloud providers provide servers that can be treated like a barebones Linux server, and that's all you really need to be productive with most server applications.
Congratulations, you’re now spending more on barebones servers than a colo and you’re still spending just as much on managing resources - now you have the worse of both worlds.
Then just use a colo center. It's worked for us for close to 20 years.
Last time someone had to perform hardware maintenance was at least 3 months ago. If you're small enough (20-30 servers for us) the things mostly run themselves. Any other overhead we have would be present if we were just using something like EC2.
The bulk of our hardware is 10 years old. We have some servers that are 17 years old and still chugging along.
> Last time someone had to perform hardware maintenance was at least 3 months ago. If you're small enough (20-30 servers for us) the things mostly run themselves.
If those servers are production servers, I'd say that's not small--you're already scaled to the point where you have hired a whole person (or two) to do server work, no?
I've worked at places where I'm the only technical person. I know enough to set up a Django server, but the reality is I'm not an IT guy, I'm a developer. Paying for a few cloud servers means we don't have to hire anyone to do server maintenance, which gives me an entire person's salary as a server budget before we have to consider whether hiring a person would be better. A lot of companies never reach the size where that becomes a concern.
Of course everybody wants to get big enough that they need a server farm, but as long as you don't vendor-lock yourself, moving from a virtual Linux server on the cloud to a physical server in a colo center is fairly trivial.
My personal website has been running on a single $10/mo Digital Ocean server for 5 years. I think it would be very difficult to beat that price with a colo center and physical hardware. Most businesses are obviously larger than that, but I think it's a bit of an overreach to claim that it's always more expensive to run on the cloud. It just isn't, for many real-life situations.
"the cloud" = your data is now our data on our computers (I think we all know this). When companies like Microsoft start saving my files to their personal servers as default, I hate you. Stop.
The appeal of the cloud is less IT politics which in big company is a serious problem. Business Units can sidestep IT and go “self service” and spare a lot of pain and time in dealing with IT. I don’t think any serious manager going into the cloud is doing it for the savings. But of course this is a bank and technology is their future so it makes sense that they want to keep a tight loop on the cloud activities.
It saved 2bn compared to what it spend on its old infrastructure right? Not 2bn compared to the (totally unknown) cost of AWS instead.
Full disclosure, I've had baml as a client and dealt with their desktop support, fix and brokerage tech support and their TRP systems. I wouldn't trust them to point out the window at a cloud let alone build and maintain one. So I'm more than a bit dubious of this story...
That's unfair - they were only fined $67 million for fraudulent overdraft fees! Though I have to wonder how much they're currently making by inducing every one of their customers to write fraudulent checks.
Though there is something comforting about a predictably evil company though. Much more straightforward than "do no evil" while building a surveillance machine that would make a Stasi officer blush, or "this helps our users" while hindering Hong Kong's protests.
For what it's worth, I really don't like AWS - aside from their S3/Route53/CloudFront products. There is a layer of abstraction that is so minimal that you may as well roll out the services yourself & save a ton of money. Of course, if you like that layer, then that's fine too.
If you look beyond that, AWS offer layers of services that you might not realize existed.
Cloud DB with no worry about physical hardware (in addition to S3 buckets which is mostly for files). lambdas when you don't want to manage a server, cloudformation when you don't want to manually start your stack by clicking, API gateway for quick ways to create APIs without worrying about stuff. And powerful orchestration layer that connects all of these together.
In fact, a lot of these services are bootstrapped on AWS itself. If you know your way around AWS, you can really find a sweet spot where it's cheap scalable and mostly worry free. The problem is that AWS domain knowledge in itself is _deep_ and not many people know enough about AWS to avoid large amount of pitfall and inefficiencies that a knowledgeable people can quickly spot and fix/alternate.
Many large companies save money by having their own cloud, but those clouds sometimes really suck compared to AWS.
I know of one very large company where any request for a change in their cloud infrastructure always required a minimum two week advanced notice. In AWS such a change is just a mouse click away, and could be done in seconds.
AWS also has a really amazing integration of a large variety of services which is really hard for in-house clouds to match. I wonder how many AWS services the BoA cloud has, and how their own integration of those services matches that of AWS.
That's not a comparison of external cloud vs. self hosted though. It is a comparison of different operating models with respect to change control. Either one can exist in both external vs. self-hosted infrastructure.
I've been in places where the engineering team plugs in an old PC under a desk somewhere, gives it a public IP address of there's the production server. That's the self-hosted equivalent of the engineering team having full AWS access and any change is a mouse click away.
I've also experienced places where any change does take a week or two of approvals even though it is hosted on public cloud.
There is a time and place for all approaches. What works best with a three person startup putting up a MVP is quite different from what is best in a very large corporation operating in a regulated environment.
Change control and service delivery time-frames are two different things. When I left BOA in 2017, you could get most production changes implemented with only two days of lead time from a formal change control perspective. But, requesting a new Virtual IP on a load-balancer could easily take two weeks, just for every layer of bureaucracy to wet its beak. And it was impossible to request something so basic without an online service request, and then follow-up emails because the standard service offerings left all kinds of details undetermined and no structured way to provide the information.
Most production changes require a second set of eyes, sometimes from a particular team, but it's all "just" code review. You put your change in the team's queue, their oncall engineer reviews it the same day, you land the change and it gets executed automatically. Most have implemented namespacing so that changes that only affect your own team's stuff can be approved within your team.
This is all on owned hardware. The difference is that we're a SWE driven company (corporate IT is off in its own world, run in the more traditional way, but they don't touch engineering's production datacenters). Infrastructure teams provide APIs, not JIRA forms.
Looks like $2B of savings are from reducing the number of servers (200,000 servers earlier to 70,000 now). That has nothing to do with the advantages of a private cloud over public, etc. They were just over-deployed and by reducing the number of servers they would have enjoyed similar savings no matter where they were running them.
In fact they could have saved even more, and earlier, if they had already been in the public cloud, where you can just spin down the VMs and stop paying for them. In your own data center, a server is already sunk cost that you cannot entirely eliminate just by shutting it down.
But the results have been dramatic. The company once had
200,000 servers and roughly 60 data centers. Now, it's
pared that down to 70,000 servers, of which 8,000 are
handling the bulk of the load. And they've more than
halved their data centers down to 23.
They say they save 25%-30% over the cloud, which is to be expected since the cloud has a profit margin. They also say that might not last.
I'm going with clickbait title. Nothing to see here other than someone getting a temporary savings which we will see as a write off in 5 years.
"Server" implies redundant power supplies, 4-?? disks, an excess of high quality capacitors, a heavy steel case, a motherboard probably 4x the area of a typical ATX motherboard, ECC RAM, far more RAM slot capacity, etc.
Furthermore, extended service contacts are the norm. Five year same day on site service isn't unusual. Five year 24/7 on call/next day replacement is typical. One year warehouse service is unheard of.
That's just the device itself. Servers also imply a rack to put them in, real estate for said racks, 24/7 HVAC for said real estate set to substantially below room temperature, and an expectation to run it (and consume electricity) continuously. You also need network architecture to support them.
This isn't even getting into the realm of stupid specced servers. 512GB RAM? Easy. 128 cores? Sure. Disk space, network capacity are cost limited: if your server is limited by network speed or disk capacity you can simply punt a briefcase full of money at your supplier and get more. You want all of that arbitrary capacity disk space to be SSD that's just an accounting problem.
320 comments
[ 4.7 ms ] story [ 267 ms ] threadThey didn't switch from AWS to private cloud.
They consolidated 60 data centers with 200k servers to 23 data centers and 70k servers.
This isn't a religious argument. Some workloads are fine on prem (known load profiles, shed infrequent traffic spikes instead of bursting), some need cloud (Netflix.com, remember all of their video content streams from their own CDN hardware), it's ensuring you're making the decision in an objective way. Sort of, but not quite, similar to making software budget and timeline estimates.
If software is eating the world, and every company is turning into a software company, why would you be surprised that orgs above a certain size wouldn't also turn into an infrastructure company? You don't have to compete against Amazon, you just need to satisfy your engineering needs for your vertical and the software that drives it. Amazon needs to be everything to everyone.
The big 3 clouds have built a nice moat and can charge whatever they want. They still compete on prices but it's clear they're colluding on network fees at least.
Oddly enough, they falter in other areas: no equivalent to Kinesis Firehose, and true to your point, I was told by customer service just to build my own DataFlow job to take a Pubsub topic and make files.
It's like some of their product managers have a very customer-first mindset (BigQuery, Pubsub) and some could not care less.
From this article: https://www.platformonomics.com/2018/05/follow-the-capex-sep...
Inside your dev team, every institutional bit of knowledge about how AWS works, how to configure it, how to work around their mostly bad docs, and how to navigate their hot garbage UI.
Every line of code that touches an AWS api.
If you need data centers in certain regions for eg localization laws, AWS is afaik the only choice.
It's scarily easy to accrue huge bills if you're not watching things closely.
They turned on encryption for an option and did not realize the huge price difference between the encrypted and unencrypted option meant $9000 per day in additional charges. After 1 month (30 days) this came to light, and he was fired.
EDIT: btw, he was not fired for using AWS, he was fired for making unnecessary costs.
Part of the organizational allure of migrating to the cloud is being able to rationalize headcount. Hybrid clouds, while bringing a lot of potential upside, are complex. While you can potentially reduce the scale of labor involved, you still have to retain all of the core skillsets you already have to maintain your on-prem locations. Then add headcount to optimize the cloud deployment, _plus_ the ever elusive skillset of the individuals that can marry the two in such a way where you show operational savings from the hybrid model.
cloud - someone else's computers
private cloud - someone else's computers that you own???? or that nobody else uses, but in this context you also own???
Private cloud usually refers to that same abstraction layer. Hardware provisioning is abstracted away from use and available on-demand from a pool, and likewise deployment works automatically via the same or an equivalent API.
That's very different from the operation of a typical company-owned datacenter from a couple of decades ago.
Traditional virtualized hosting environments typically also come with varying levels of technical overheads when setting up new boxes.
What bofa have is a virtually self-service (some business approvals needed) user interface with the ability for devs to set up new virtual servers with little/no knowledge of the underlying hosting infrastructure.
A billion dollars sounds like a ton of money on HN, but that's a thousand person tech team for 3 or 4 years. It's enough to do some good tech, but it's very, very easy to spend that much engineering capacity.
There's definitely financial savings to be had by doing it in-house (either literally or utilizing existing data centers as here). But what I've experienced is that companies are often attracted to Big Cloud™ not just for financial reasons but to flatten/simplify their corporate reporting structure. It is the ultimate delegation.
This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously).
These "soft" costs/savings are rarely discussed because they cannot be measured, unlike a balance sheet. But there's definitely a whole other dimension to infrastructure outsourcing that is worthy of note.
In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks.
There is absolutely value to delivering a product to market a year or more sooner than otherwise.
I’ve encountered quite a few devs whose only focus is on achieving the specified goal without considering the full security scope. I don’t blame them, that’s what they are there for. Stepping in for ensuring the customer security and regulation requirements is what I’m here for.
It's like using package managers like Maven and npm. Sure, they make going to market much faster for small organizations. But at companies that have strict security procedures, downloading unvetted stuff from the internets and putting it into production does not fly. You'll have internal repos that work with your package manager tools, but nothing gets into those repos without at least a cursory review.
The two companies I've used AWS at both operated this way. In fact, each team has multiple product accounts (for dev, testing, production). So I don't think it's unusual at all.
Separate from all of that there is a valid question whether the software stack at a big cloud provider gives your engineering department the ability to deliver features faster at scale and redundancy, and with lower ongoing maintenance overhead. I think this question is very much up for debate.
At a small scale there are absolutely cloud frameworks that let you deploy services without an IT department at large scale. But then there’s a realistic argument of whether the high premium for theoretical scale—when in actuality you’re serving dozens not millions of customers—is worth the high cost of capital.
E.g. several startups of mine have persistent on a couple colocated dedicated servers for over a decade at a cost of $300/mo. The equivalent deployment in the cloud is closer to $3,000/mo.
I know this because I got $300k of Azure credits to play around with and ended up not using most of them. The long-run cost was too high to move to Azure. But there were months when I “spent” $6k of credits on a dozen VMs and associated storage and network that had equivalent performance to my rented bare metal.
I think the opposite will happen. Cloud providers are becoming commoditized. Kubernetes, blah blah. Developers hate lock-in, and on a long time scale, developers drive tech decisions. Nowadays new software is cloud agnostic, and it will continue to be that way. This is commoditization, which generally drives down prices.
That said, buying commodities from a market with four participants is more like buying from a cartel than a marketplace. I definitely worry about the consolidation of the world’s IT spending going into the coffers of three or four massively profitable and morally corrupt corporations.
You and enterprise software should get together and have a introductory meeting...
I think some companies, like BOA, look at this as the safer alternative.
We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world conflict. These global companies with outsourced IT will be in ruins because politics and digital DMZ's will keep them from acting cohesively.
For an organization like a bank that's worth over $400b that layer of staffing, equipment, oversight, and all the inefficiencies are a $400b insurance policy that you will be able to continue operating regardless of geographic or political events outside of your control.
It was about decision maker's personal (career) risk when making decisions. Keeping things in-house means more decisions are taken in-house which increases personal risk to the management for getting things wrong. In context this is much clearer:
> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously).
The post above doesn't discuss cloud Vs. non-cloud security/safety at all.
I've been at multiple Fortune 50 companies at multiple levels. Many times these types of infrastructure building are just forms of empire building and an attempt to embed ones' services in the organization at a very deep level. You convert one set of costs (AWS/GCP/Azure bills) for another set of costs (Infrastructure + Payroll). You also convert one set of headaches (CSP overcharging/auditing) for another (prima donna internal infra engineers.)
Don't get me wrong, most corporate IT is deeply incompetent at their core job. It's just that you can't fix the deep problems by outsourcing the computer part.
There's this great thing about BOA-sized support contracts where you get to call up the senior product managers and demand that they build the features that you want. And if you're not happy about something, engineers will drop what they're working on to appease you.
I wonder how many customers were interested in setting up a Custom Keystore for KMS on CloudHSM?
https://aws.amazon.com/blogs/security/are-kms-custom-key-sto...
Of course, for small or medium customers whose finger snaps are ignored... WYSIWYG. The roadblocks in not-your cloud are still your roadblocks.
But (in a vacuum) when developing a solution to serve diverse needs and stakeholders, the HiPPO effect is definitely an anti-pattern.
With CSPs, at least you have four major players (AWS, GCP, Azure, Oracle Cloud, ?IBM?) competing prices downward and multiple theoretical options. Yes, you have your in-house CSP expert biasing things, but since most of the work is external, at least you dont have a heavy internal weight on a home-built choice.
IMHO, the better option would be a CSP-agnostic Kubernetes based layer. But IMHO we're not there yet.
To your point, we need only point to recent events to show the universal customer pain that can be caused by a bad code push at Azure or AWS.
If a company (or person) really wants complete visibility and control into their relationship with law enforcement, they have to host their own hardware... ideally, their own data center.
I've always wondered about this. In IT now instead of having 5 system operators one has 10 "cloud consultants" and nobody knows what they are doing. I've been through a couple of "migrations to the cloud" now and I've never seen a reduction in IT staff or the cost of running IT operations.
About the wrong decisioning part, if you chose the wrong "cloud" for your business the cost to change can be enormous.
The primary motivation for management is from what I've seen so far the ability to point to someone else if shit hits the fan, not cost reduction and all that other stuff.
You start wondering if 90% of corporate world are there to do nothing if not erecting roadblocks for people who are actually doing something. But then you start wondering maybe it's a good thing that all of these people get paid and could spend their money so that majority of the country has purchasing power and not live in piss poor poverty.
This is a myth directly from cloud providers marketing campaigns: that using the cloud simplifies infrastructure to the point that you don't need as much and/or as qualified staff.
In reality it is quite the trap: without experienced system administrators AWS and the like are free to dig their hands deeper and deeper into the pockets of their unsuspecting customers.
Believe it or not most U.S. programmers don't live in the Bay Area.
I'm surprised that more companies aren't playing moneyball with nice, but cheap, locations.
Tooling has only improved.
We were able to cram a large set of applications into a group of instances using CapRover.
Instead of 2-3 instances, and a whole bunch of RDS, we ended just recreating the databases there along with minio for storage (we write a small amount of data to S3 storage).
Colocation is probably ideal for anyone using custom hardware: like GPUs or FPGAs. If you're using "normal" CPUs, just buy dedicated instances instead.
You can rent vs buy at any level : rent only machines, rent a 1/2 a rack, rent a full rack, rent a cage, rent 1/2 a dc, and so on.
Same with connectivity: you can plug your machines into a lan managed by the hosting provider, or run your own routers and peer with their in-house ISP, or you can buy peering directly in the building, or you can rent lightpath from a telco with pop in the building, or you can go rent a backhoe and start trenching across the parking lot...
Anyone who has been in the industry since pre-cloud explosion knows that most people just rented bare metal and at the more complicated end they might purchase their own hardware and colocate.
And for those that own their servers, many lease them and HP or Dell or whoever maintains them on site if anything goes wrong.
Maybe cloud usage can be an indicator whether a company has plans to maintain independence into the future, despite the loyalty-engendering words of leadership.
I mean. You save the headcount in one area and you consume it in another. And you pay overhead in outsourcing costs.
It’s likely worth it. But you need to determine that on a case by case basis.
My (large) company is too incompetent to do hosting properly so cloud was a blessing for us.
At a certain scale, you need all these things with internal clouds too. Resource provisioning, basic security, and managing "insider risk" don't just go away when you own the infrastructure.
This is like saying that "You also need to have a cashier!" when you're comparing a shopping mall to a corner store.
I think it's a loooot easier to design and build a cost-efficient multi-tiered high availability application than it is to make reasonable cost projections in whatever micro-currency every seconary and tertiary service uses.
And I'm ever surprised by how time-intensive it is.
Cloud is awesome for companies where the capital investment needed to deliver the SLA is too expensive.
If you’re big, and you run the numbers, lots of services are better in a datacenter that you manage. Many of the early advantages of cloud tools aren’t a differentiator today. If anything, the granular billing is a nightmare to manage in many large enterprises. There’s a reason AWS heavily targets .gov customers — the Feds probably waste billions on idle services.
I’ve found that the shitty to support or known workload services are best somebody else’s problem. Email is an example of both. Most other things are trading IT complexity with accounting complexity.
I agree the cloud providers like this idea, but I'm not sure I agree it's a myth. Businesses need to choose what competencies to focus on. For nearly every other area, businesses rely on suppliers. Auto manufacturers use machines built by other businesses and parts built by other businesses. They also obviously do some amount of in-house part and tool design. It's a balance and I think that it's a lot more likely that most businesses will want a good partner to provide infrastructure (ultimately at margins less than AWS) instead of building it themselves.
Yeah but they generally aren't renting those machines. Renting critical pieces of business infrastructure that aren't easily replaceable is opening yourself up to rent seeking on the part of your supplier. Remember the Oracle business model--lock people in and then keep raising the price just below what it would cost to rebuild from scratch.
In other examples, many airlines do lease airplanes, and even among the airlines that do own the planes, they still usually lease the engines.
As the GP said, there's a balance to be struck. Delta doesn't have the expertise or the economies of scale necessary to build their own engines from scratch, but they do have the ability to lease those engines and then have their own teams maintain them. Similarly, most companies don't have the know-how or the EoS to run their own data center, so instead they buy cloud services and then have a smaller team to maintain it. It's a win-win. Some companies like BofA might be big enough where they can run their own cloud and it be cheaper than using a cloud provider, but that's definitely not a common thing.
Rental only makes sense if there is a marketplace for used machines. Eg, planes and plane engines are components that aren't really tied to one place or company. (The "that aren't easily replaceable" from the parent post.)
But industrial robots for car manufacturing seems like rather specific hardware. If, say, the Ford plant in Claycomo, Missouri no longer needs a machine for making the F-150, who is going to rent it? How is rental a better cost-savings for Ford than buying the machine and later selling it?
I tried looking this up, but failed. The closest I found was that "over half of industrial robot purchases in North America have been made by automakers" at https://www.robotics.org/blog-article.cfm/The-History-of-Rob... .
Also, '"rent" the maintenance contracts' doesn't make sense to me. I sell maintenance contracts to my customers, I don't rent them, and I don't understand what "rent" could even mean in that context.
The parent comment mentions that Delta rents their airplane engines. I imagine that they have the capability to do general maintenance on the engines, but that the engines need to be periodically sent back to the supplier (probably GE or Rolls-Royce) for a teardown, safety inspection, and rebuild. Without regular inspections an engine can't be flown, and is essentially worthless. Not much point in actually owning the engine if you can't use it.
With that in mind, I can see support contracts as an operational expenditure.
But if "over half of industrial robot purchases in North America have been made by automakers" then that's a lot of capital expenditure. I never got the feel that the companies which make modern industrial robots primarily rented them out (akin to IBM's renting out of tabulation machines 100 years ago). I really thought they mostly sold the robots.
An assembly line isn't really a "product," per se. It's more of amalgamation of several products and technologies, many legacy, that forms a cohesive but loosely-coupled whole. You may have several vendors' products / systems in a line. So what you end up with are ongoing contracts that provide for installation, programming, troubleshooting, maintenance, etc.
There's actually a whole specialized industry that does nothing but design assembly lines. It's pretty neat.
There's Boeing which also used to have an airline. It was made illegal IIRC.
And there are things in between these two extremes.
Some airlines have a heavy maintenance organization. Some have outsourced it. Different companies have different market segments and safety records etc.
The point is, there's no one true way for all.
Of course it's a question of scale, but a lot of arguments I've seen usually come down to CapEx vs OpEx accounting which is silly because a dollar is a dollar.
So no... not necessarily... CapEx is not necessarily paid out all at once. So the accounting ought to be the same whether its a CapEx purchase under a Subscription Agreement/Loan, or if its an "OpEx" payment for 1 month of services.
The cloud is powerful, but for long running businesses who can afford to invest in the competancy it's stupidly expensive. Hybrid cloud installations are ever more the norm, private clouds increasingly capable (PaaS, IaaS, SaaS), and toolsets like Kubernetes reshape how a lot of those legacy apps get looked at.
Situation: some manager is tasked with updating some antiquated COBOL monstrosity. They're given twelve peanuts, some shoe string, bubble gum and a little bit of duct tape. The experienced staff have all been aged out, quit out of frustration or laid off and replaced with fresh college grads for half the price which was a great deal and significantly reduced costs for several consecutive quarters, earning a well deserved healthy bonus. Unfortunately reliability of the system has been suffering and the young engineering team unanimously agrees the old system should be replaced.
Option 1: do it in house with the young team of rockstars the manager has been bragging to management about. Unfortunately, the manager secretly has reservations and has doubts about the team's experience, let alone skills. It's too risky. Not too risky to the company who can easily absorb the loss and move on, but too risky for the middle manager who will look bad and possibly get fired.
Option 2: outsource the entire thing to IBM or Oracle. Tempting, but not in the budget. Also, significant parts of the old system are on IBM anyway and the cost of hiring IBM people to fly out has been increasingly expensive after the latest round off layoffs saw Scott put out to pasture. No, it's too risky. Not risky to the company who can have uptime guarantees in the contract, but risky to the manager who will look bad if he comes in over budget.
Option 3: The Cloud. It's the 'in' thing. FAANG are all all in on it, and if it works for them it could work for us. Much of the new team used AWS for their web dev class in college, so the skills are fresh. There's very little risk to it. The staffing requirements will be dramatically reduced so costs will drop. The manager will be able to deliver an awesome demo next quarter and the S3 instance they'll host it on will only cost one or two of their peanuts. The only risk is if the manager doesn't get promoted before the problems start, which shouldn't be a problem because AWS has such great onboarding programs. It is slightly more risky for the company though, who may find themselves no longer in control of critical infrastructure.
Risk for the company isn't the same as risk for a mid level decision maker. Small fuckups carry the same penalty as huge fuckups. (you get fired) From my fake story, doing it in house has a relatively high chance of being a minor fuckup, and AWS has a relatively low chance of being a major fuckup. The manager is going to choose the low probability risk, even though the low impact risk is the company's best interest.
And let's be honest... a remarkably large fraction of small IT shops are really bad at certain basic competencies, notably backup reliability and timely application of updates. AWS is pretty good at sysadmin and accounting 101, even if it stumbles at 201.
Insightful. Can you clarify what you mean by "low probability risk" and "low impact risk" ?
impact risk - Odds of a project failure causing large impact to company.
There are two things worth considering. The BOA cloud is slow. That is to say, it is hard to procure machines to get your projects going. There is a lot of control on who will be paying for them. There is also A LOT of staff to manage it. So those 2 Billion could easily be tweaked downward when you consider the staff and lost of agility.
On the flip side, AWS is hella expensive. To do the things that they do it would cost A LOT of money. Maybe even more than they think. I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills. So this number can also be tweaked up when you consider dozens of teams each having their way (at BOA scale) on the AWS console.
But I would expect that a large established company would do better to create it's own reasonably priced reasonably managed infrastructure.
Sort of rent vs build/own.
And something that seemingly everyone forgets: it's not black and white... you can have some things in the cloud while you have some thing on-prem, leveraging scalable infrastructure and redundancy while also keeping your core assets fully under your control. It's pretty easy to make an internal S3 replacement, it's hard to make a better S3.
A company like BoA will eventually choke until this in the future.
Also, this stuff sounds great until you need improved peering, or software-defined networks or massive bandwidth improvements, etc.
... and I get that the HN community is clever but banks and other institutions like this REALLY struggle to find this talent.
$2B is a lot except when you're losing $10B a year in revenue because you can't compete in the market.
In a way this would be like Amazon arguing it should build its own bank. It's not what they're good at... Focus on your strengths.
For example if I can project steady growth or decline of my needs, it seems that I benefit less from the infrastructure-on-demand like AWS/Azure/GC
Also, depending on the size of the organization it seems that there at least several options on data centers:
AWS/Azure/GC are in 4th category, but cost savings in terms of electricity/building capacity/peering points are available in the first 3 as well.Many organizations deploy Pivotal's https://pivotal.io/industries/financial-services
or Redhat/OpenStack
So they are not actually 'building' the infrastructure management layer from scratch.
Also for investment banks (due to exchange connectivity) , telecoms (due to peering points, physical tower locations) always going to have their 'Front-office' applications in data centers that offer lowest latency for those apps.
Midoffice applications (or BOSS as they are labeled in telecom) can live in data centers reasonably distant (in terms of ping time) from the front office. So the infrastructure for those has a different cost profile.
Most of this applications touch trades or trade events that carry millions of dollars worth of contract value (not Facebook's likes or favorites... )
There is a lot of logging, reconciliation, post executing enrichment that's probably going on.
Finally, from my 2nd hand understanding,
Investment banks can have from 3K to 10K applications in production. With, perhaps 20% of them having more than million lines of code.
Some of those apps are built in their own proprietray languages developed specifically to manage contracts and lifecycle of contracts (Banks had 'smart contracts' DSL and interpreters since 90s).
https://news.efinancialcareers.com/uk-en/282097/slang-goldma...
- Access to cheap real estate
- Access to enough human capital, especially a team of 80+ to operate all aspects of a cloud: hardware, switches, openstack / kubernetes
- Requires scale of 20k+ servers = 320k+ VMs and a strong inclination to not host on AWS/Azure/Others.
- Lease to servers costing about $250 / server / mo. With AMD, you could go half of that, with VMs costing $10 a month. Add network costs of $15 / 2 ports / server / month.
It's not like Amazon doesn't know this too (e.g. they try to make the transition harder by offering lots of proprietary services that help you ramp up faster, but can't be used elsewhere)
There are plenty of reasons to choose a point in the space denominated by the axes make/adapt-existing X insource/outsource X proprietary/open . None are zero cost unless you really think your platform is good enough that enough others will write for it without you doing anything.
It's possible that you would chose to write your own because nothing in the market is even close to good. It's possible that you would open source it in the hopes that it becomes standard and that others will add new features that you will benefit from too (this is not a zero-cost option either). It's possible that you would even keep it proprietary because that's cheaper. But given the way Amazon operates it's pretty clear that they offer their own proprietary services to lower the barriers to adoption ("simply offer a better option") and raise the barriers to switching.
I don't see anything wrong with this strategy legally or morally, though I mostly avoid amazon proprietary tools because the risk of locking is one I'm not willing to embrace. There are of course situations where they aren't a problem (e.g. a tool with an anticipated finite lifetime: the cost of lock in in that case is essentially nil).
Were the opposite true, you'd wonder about the economics of the cloud business model.
I think a more interesting way to look at this is, are the other advantages of cloud services (e.g. lots more flexibility and bundled proprietary technologies that you can't economically build yourself) actually worth the extra money?
Yes, by not using a cloud service you might save $2B a year, but does that cost you the opportunity to make even more than that, given you're probably moving slower or at least less efficiently than you otherwise could?
The potential opportunity of fast-moving cloud features needs to be weighed against the opportunity costs of slow-moving cloud features. Where bespoke solutions can immediately provide tailored performance and maximize technical capabilities, a missing feature in any of your cloud providers services can be a showstopper or unmitigatable roadblock. And while lots of the technology is past the bounds of reasonable economical replacement, some of the technologies being shared through the cloud are nigh unfathomable to recreate.
Which is to say that the black ju-ju behind Windows update probably takes making a new MS to build up to present maturity, and unless you're a certified "big boy" letting small teams somewhere else fully dictate what you can and can't do at service boundaries probably impacts you in the long run.
Based on that, and IMO/IME: the answer isn't a binary choice but a constantly shifting point on a spectrum between the two, where on-premise/local-cloud and remote-cloud services are aware of one another and maximize capabilities while minimizing costs. Hybrid installations are just stronger, and are easier to reshape according to costs.
I wonder what the other 62000 servers are doing.
That being said, we control the HW and SW stacks end to end so I don’t have to worry as much about the nightmare scenarios the NordVPN folks went public about today. Critical given we’re in fintech ...
So much of the Fortune 500 lust for public cloud seems to come down to working around inefficient IT procurement and provisioning processes. They haven't automated, they don't maintain enough excess capacity, they haven't managed vendor relationships to assure fast order turn-around, financial controls are too onerous, etc.
Everything's virtualized but otherwise they're still operating like it's 1995.
I am glad they shared this situation. We often here marketing telling us that public cloud is always better. It's nice to see different perspectives being shared and some details around them.
It is worth doing this if your in-house engineering team is at least 100+ app service developers and have high feature churn rate.
But be warned, your systems engineering / shared technology team should be level-headed and mature and rest of your app service engineering should be good too to pull this off well. If not, you will be in serious developer productivity pain and there won't be quick and easy fixes once you have put down the capex.
The cloud is valuable because of the variety of features, immediately available, at very large scale, run by very credible people with $0 in capital expenditures.
In most cases, it's worth far more than the price, which is why it's so popular.
In very, very few cases is it 'cheaper' to run one's own cloud given the massive operational overhead of hiring people with very specialised skills, risks all over the board, the enormous planning, extensive investments.
Wallmart must have 1000's of trucks but they don't 'roll their own' delivery vehicles, because there's an inherent degree of specialisation in making vehicles.
So yes - in some specific cases it makes sense to 'host': say Dropbox, just wants massive cheap storage, or Google, has special needs & scale & ability, or some entity has very specific privacy requirements - and of course I wish there were more variety in providers, especially localised ... but the cloud is essentially now the default hosting option and that's not going away because it makes a lot of sense.
The surpluses generated by such services are vast - it's a big step forward for everyone. Companies using AWS are probably profiting more in surplus than the margins in AWS.
For many companies, hiring and talent retention is a problem. There are some risks that are not as superficial as dollar signs on the balance sheet.
Last time someone had to perform hardware maintenance was at least 3 months ago. If you're small enough (20-30 servers for us) the things mostly run themselves. Any other overhead we have would be present if we were just using something like EC2.
The bulk of our hardware is 10 years old. We have some servers that are 17 years old and still chugging along.
If those servers are production servers, I'd say that's not small--you're already scaled to the point where you have hired a whole person (or two) to do server work, no?
I've worked at places where I'm the only technical person. I know enough to set up a Django server, but the reality is I'm not an IT guy, I'm a developer. Paying for a few cloud servers means we don't have to hire anyone to do server maintenance, which gives me an entire person's salary as a server budget before we have to consider whether hiring a person would be better. A lot of companies never reach the size where that becomes a concern.
Of course everybody wants to get big enough that they need a server farm, but as long as you don't vendor-lock yourself, moving from a virtual Linux server on the cloud to a physical server in a colo center is fairly trivial.
My personal website has been running on a single $10/mo Digital Ocean server for 5 years. I think it would be very difficult to beat that price with a colo center and physical hardware. Most businesses are obviously larger than that, but I think it's a bit of an overreach to claim that it's always more expensive to run on the cloud. It just isn't, for many real-life situations.
Full disclosure, I've had baml as a client and dealt with their desktop support, fix and brokerage tech support and their TRP systems. I wouldn't trust them to point out the window at a cloud let alone build and maintain one. So I'm more than a bit dubious of this story...
Though there is something comforting about a predictably evil company though. Much more straightforward than "do no evil" while building a surveillance machine that would make a Stasi officer blush, or "this helps our users" while hindering Hong Kong's protests.
Cloud DB with no worry about physical hardware (in addition to S3 buckets which is mostly for files). lambdas when you don't want to manage a server, cloudformation when you don't want to manually start your stack by clicking, API gateway for quick ways to create APIs without worrying about stuff. And powerful orchestration layer that connects all of these together.
In fact, a lot of these services are bootstrapped on AWS itself. If you know your way around AWS, you can really find a sweet spot where it's cheap scalable and mostly worry free. The problem is that AWS domain knowledge in itself is _deep_ and not many people know enough about AWS to avoid large amount of pitfall and inefficiencies that a knowledgeable people can quickly spot and fix/alternate.
I know of one very large company where any request for a change in their cloud infrastructure always required a minimum two week advanced notice. In AWS such a change is just a mouse click away, and could be done in seconds.
AWS also has a really amazing integration of a large variety of services which is really hard for in-house clouds to match. I wonder how many AWS services the BoA cloud has, and how their own integration of those services matches that of AWS.
I've been in places where the engineering team plugs in an old PC under a desk somewhere, gives it a public IP address of there's the production server. That's the self-hosted equivalent of the engineering team having full AWS access and any change is a mouse click away.
I've also experienced places where any change does take a week or two of approvals even though it is hosted on public cloud.
There is a time and place for all approaches. What works best with a three person startup putting up a MVP is quite different from what is best in a very large corporation operating in a regulated environment.
This is all on owned hardware. The difference is that we're a SWE driven company (corporate IT is off in its own world, run in the more traditional way, but they don't touch engineering's production datacenters). Infrastructure teams provide APIs, not JIRA forms.
In fact they could have saved even more, and earlier, if they had already been in the public cloud, where you can just spin down the VMs and stop paying for them. In your own data center, a server is already sunk cost that you cannot entirely eliminate just by shutting it down.
I'm going with clickbait title. Nothing to see here other than someone getting a temporary savings which we will see as a write off in 5 years.
"Server" implies redundant power supplies, 4-?? disks, an excess of high quality capacitors, a heavy steel case, a motherboard probably 4x the area of a typical ATX motherboard, ECC RAM, far more RAM slot capacity, etc.
Furthermore, extended service contacts are the norm. Five year same day on site service isn't unusual. Five year 24/7 on call/next day replacement is typical. One year warehouse service is unheard of.
That's just the device itself. Servers also imply a rack to put them in, real estate for said racks, 24/7 HVAC for said real estate set to substantially below room temperature, and an expectation to run it (and consume electricity) continuously. You also need network architecture to support them.
This isn't even getting into the realm of stupid specced servers. 512GB RAM? Easy. 128 cores? Sure. Disk space, network capacity are cost limited: if your server is limited by network speed or disk capacity you can simply punt a briefcase full of money at your supplier and get more. You want all of that arbitrary capacity disk space to be SSD that's just an accounting problem.