IMO, an ideal co-op cloud host wouldn't have a bunch of complicated p2p junk. Just a good ol' datacenter and people to operate it, with the cost and governance shared by the organizations running workloads on it.
Yes, I do; sorry about that! They seem relevant, a lot...
I was on the development team for a couple years; now, I'm independently developing some technology on the platform. I'll announce it soon, I hope.
With the recent AWS outages, and systematic de-platforming of some legal businesses, there seems to be a growing realization that dependence on centralized hosting platforms is ... dangerous.
Holo / Holochain seems to provide a unique solution to some of these problems.
pure FUD. bugs happen decentralized systems can be even more problematic to resolve issues in due to their decentralized nature and the usual requirement that fixes be deployed system wide.
The other direction is to use the commercial clouds but aggressively defending their status as a commodity. Have VMs at multiple providers and move your load between them as needed.
This. Turn the "cloud is somebody else's computer" on its head and own part that computer.
Why not "own" AWS, GCP, Azure etc.? Because you can't. They are part of conglomerates that come with a moral baggage that is too heavy for some.
ps. No "hyperscaling" ability required. The idea that business activity is either complete dominance or death is part of that unfit-for-purpose mentality.
its not exactly the same. there are probably significant efficiencies (energy, security etc) in provisioning and sharing "cloud" hardware in reasonably large chunks.
> Akash Network leverages 85% of underutilized cloud capacity in 8.4 million global data centers, enabling anyone to buy and sell cloud computing. [0]
How do you even begin to operate a consistent security model across "8.4 million global data centers"?
> Auditors on the Akash Network review cloud providers and digitally sign the provider on-chain with their certificate. If you only accept bids from audited providers this means you are trusting the Auditor/Provider not just a Provider. [1]
For a business at scale, I don't see this being solid contractual ground. Now I have to trust Auditors to continuously vet Providers? Simply too tenuous for my comfort.
An article that regurgitates Gartner talking points is a tainted one to begin with. Gartner even famously said that Apple should quit the hardware business:
Twice in my career I have been involved in evaluating enterprise products for a large corporation - and part of that process was looking at Gartner reports.
The Gartner reports were complete and utter garbage, filled with factual errors. Looks like they were written by interns, regurgitating marketing material - completely removed from reality.
Since then, whenever I see somebody mention a Gartner report, I immediately dismiss that part of the argument.
Can confirm. Was a developer for 20 years, got an MBA 20 years ago and have been dismayed by the quality of Gartner reports. Worthless for anyone with experience, and dangerous when used to guide the non-technologist. (Yes, I've been developing for 40+ years.)
But mgmt LOVES Gartner reports. It helps them avoid/shift blame when a purchasing decision goes awry. The first question our VPs ask about a new technology is "Is this %VENDOR% in the Magic Quadrant?"
I am convinced that Microsoft underwrote them to boost Windows and cast FUD at everything else in the 90's, and they've never recovered their disingenuous ways. What annoys me most are the market share reports, implying that Microsoft owns the entire computing world. They don't. For personal use, I'd say Apple has broken into the majority. I'd pay money out of my own pocket to see PC vs Mac sales numbers with corporate purchases removed, rather than the same report they've published every quarter for 20 years now.
Gartner has a long history of putting out garbage. Some called them anti-Linux, but I was always in the camp that they were just writing crap for large quantities of money and the people paying wouldn't know the difference. This story from 1999 is surprisingly still up: http://www.cnn.com/TECH/computing/9910/22/gartner.linux.idg/...
> The second report ("1999 OS forecast: The Linux face-off") projects market share in server dollars for the next few years. This one even contains a pretty graphic. The Gartner Group finding is that Linux will fade from the scene following the release of the first service pack for Windows 2000.
> The third report ("Red Hat's future: Boxed in") concludes that Red Hat will fail to overthrow Microsoft by 2004, and suggests that Red Hat needs to find a more viable business model.
If you get a report from them on how the pitfals of setting Oracle's Spatial, it's probably good. If you get a cost benefit comparison of Oracle and Postgres, it's certainly trash. (Who am I kidding? They still insist Postgres doesn't exist.)
That said, none of the documents are worth the price you pay and the hassle of interacting with them.
Basically analysts mash up what vendors and customers tell them what they think. They have conferences with large enterprise customers and vendors sponsor those conferences. So there’s a element of indirect pay to play. Many companies use them to review contracts as well, so they know what kinds of deals are out there.
If you expect to understand technology from Gartner reports, that is dumb. But if you want to understand the market and offerings for large commercial/gov enterprise customers, they are a great resource. If you are a startup, you should not pay any attention to Gartner unless you are selling in that space.
In the case of this topic, the information that’s key to understand is that Gartner is getting feedback that their customers are adopting cloud services, but are seeking to optimize the spending, including with some of the smaller players in niche areas.
> But if you want to understand the market and offerings for large commercial/gov enterprise customers, they are a great resource.
They are pretty shitty for that too. They only get anything passable if you are after the offerings of the specific companies that pay them for the publicity.
>Ms Wang and Mr Casado have suggested that firms should think about building their own private clouds to keep costs down
Yeah I distantly remember "on prem", too. Does everyone, everywhere really need to be equipped to hyperscale at a moment's notice? Wouldn't it be better not to be beholden to a megacorp that serves you with 30 pages of opaque billing for business-critcal functionality? Doesn't hiring external consultants to manage your fucking cloud infrastructure seem like sort of a risk as compared to keeping the services-wrangler in-house?
...I don't have an MBA, so these are half-serious questions. If anyone can chime in with (at least) a half-serious answer, I'd halfway appreciate it. I understand that The Economist is the voice of global capital, and global capital has decided that there's money in them thar clouds, but still!
I have worked with hundreds of companies moving to AWS, and believe that this is the answer.
People are investing in cloud because its faster, more agile and better, not because it's cheaper.
The top line looks expensive, but if you then do the correct analysis on the cost side of the equation (taking out staff costs and DC costs) then I think it is either close or in the black from a cost perspective.
If the business case was really bad, I don't think we would see the rapid uptake we have seen.
If you think from an economic standpoint, Amazon should price AWS only a bit cheaper than the on-prem options. It's a way to maximize their profits.
Then you have competition (GCP, Azure, Alibaba) which will tend to lower the price, but for now the cloud industry grows. Their goal is to attract new customers who didn't use the cloud before.
have developed on both AWS and non AWS server and db. There is no deference now days. Depending on if you want to hire the right people with the right skills is another thing.
Also experience. My team has been doing cloud deployments for the last 18mths, now we are on a gig at a company that has onprem. In theory the onprem infrastructure is comparable to cloud (K8's, GPU's, HDFS, Kafka) but the reality is that to get a set of K8's pods for a release has taken 2 weeks, 10+ meetings (calls) countless messages and 40+ emails.
And the ones we got on Friday were wrong (we asked for 3 * 16GB ones + 1 * 2GB, we got a 16GB project container with 4GB pod limits).
For our hadoop request - yes it got provisioned in >8hrs (mostly I think because no one is using it) but none of the access instructions work and everyone who has used it is mysteriously unavailable - so we can't access it.
Compare and contrast to self service from any cloud.
Why : well it's demand management. If you do self service and say "pay for what you want" then IT's costs bloom and the CIO is shot. If you do that in a cloud scenario then Cloud costs bloom and the CIO says "you guys have spent your money badly". This means that you need police and you need people who you ask to give you what you need if they are happy. These people obstruct you and then get things wrong.
You bring up some great points. There are two that I see.
First, there are internal processes. All too often they end up needlessly slowing velocity. This isn't a technology problem but a people problem. Using a public cloud enables people to circumvent these issues, in some cases.
Second, there are costs. What does it cost to give someone the features they ask for in the time they want it? This is a requirements, design, time, and cost problem.
Basically small markets (internal IT) can't sustain the investment to support the requirements design time cost issue - while the public cloud folk can say "any colour you like so long as it's black" and there are enough takers to pay for it all.
I think this is a common need, even in smaller organizations. Spinning up/down deployments doesn't imply you're doing it in response to scale needs (most orgs don't have those); often it means you're simply experimenting with new tools or new product offerings. Having the ability to do that without waiting for infrastructure provisioning, and having the ability to only spin up the resources you need to see (for example) if a given version of a web framework can run your site well, or of a new database meets your needs, can be a significant value add.
> often it means you're simply experimenting with new tools or new product offerings.
That only makes sense to me in the context of a business being an IT shop to begin with. Most businesses aren't, and their new product offerings require factories and dies and labor and materials. Besides which, the problem of provisioning is entirely overblown for this scale of business. I can spin up new servers all day on our on-prem vSphere cluster.
sense to me in the context of a business being an IT shop to begin with.
Lots of businesses outside of 'IT shops' use computers. I currently work in Civil Engineering and more and more of our product and service offerings have an IT/Online/Web based component which involves setting up some sort server.
I can spin up new servers all day on our on-prem vSphere cluster.
I what price? At my old job an on-perm vSphere 'machine' cost my department a lot more than what AWS was charging.
We've had the same vSphere cluster for 5 years now. I forget what it initially cost, but over the timespan in question I think it would be very difficult for Cloud-anything to cost less. Hell, the licensing for the software we use to do business costs an order of magnitude more.
Fortune 500 companies have hundreds of internal apps that each have their own stack. Multiply 2 or 3 for lower environments.
At that scale in a single company the ability to provision verticals weighs heavily in architecture and design choices.
"Should we spin this new feature out as a microservice or fold it into an existing app so we don't have to wait weeks and do crazy cost accounting for new environments?" And the answer has follow-on effects because now it has to be accounted for in a specific teams' backlog and we encounter yet more waiting.
I work in a very boring industry very far away from SV and spinning up and down machines for doing calculations and analysis is a godsend. I have a 'cluster' of 6 machines with 128 GB of RAM each set up and ready that I can start up with a script and only pay for the less than 200 hours a year I actually need them. Sometimes however I need 256 GB of RAM so I just change a parameter in my script and, magic, I have 256 GB of RAM. For other workloads the optimal setup might be 100 1 core machines with 2 GB each. So I type some commands, and now I have that instead.
And since it's my 'private' cluster I don't have to worry about queuing my jobs and waiting for the machines to be free. There is no way my department would have bought me those machines as physical hardware.
It's a need born of our tooling. The only way to stay sane is to operate extremely simple services, or to put every service in its own container (or VM) that can be torn down and re-created automatically. The result is a need for one or both of: excellent container deployment tools; fast & easy ways to spin up "a new machine" (a VM) on demand.
People also seem to be overlooking the benefits of downward optionality - you effectively transform a fixed cost (servers and staff to run them) into a variable one, that you can flex down cheaply (no long-term leases or redundancy payments). It lets you test things out (e.g. entering a new market) and cheaply wind them down if they don't work out.
I've mentioned it before and will again: AWS spent a decade openly and privately advertising how much "cheaper" the cloud was. How economies of sclae meant you'd save boatloads of money. Except, it was never true. You will potentially save money if you're just starting out and have no idea if your idea will be successful. A massive capital outlay is difficult and stupid at that point. You will also potentially save money if the tool you use to make money is something you can easily scale up and down, and actually DOES scale up and down. IE: I host a fantasy football stats app and every Monday night, Thursday night, and all day Sunday I've got millions of users, and thousands the rest of the week.
If you don't meet those criteria, unless you are absolutely horrible at negotiating with your vendors, you aren't saving money going into the cloud. I don't care that "datacenters are expensive to build". If you are large enough to need to build your own datacenter instead of renting floorspace for pennies on the dollar from Switch or Equinix (although Equinix is kind of pricey) or Windstream or insert provider - then the datacenter is a rounding error in your 10-year IT budget.
The problem is "I moved company XYZ to the cloud" is now a requirement for an IT exec to show up to the country club on Saturday. So whether it makes any fiscal sense at all, the board wants to know "what our cloud plans are" and most execs are more than happy to oblige in order to pad that resume. In my experience, most of them leave long before the bill comes due, and people start asking why the IT budget quadrupled in 3 years. The best part is, most of these companies can't pass along that increased cost to end-customers like a Netflix can because they aren't actually making money on the apps they're hosting in the cloud, those are all internal business facing apps.
Money is ALWAYS the issue. Whether you can pass along increased cost to your customers or not is entirely dependent on what your business is and the competitive landscape of it. I don't care how "agile" the cloud makes you, if your cost of IT is more expensive than the revenue you bring in, you're bankrupt. It's the running joke around here about people who build autoscaling k8s based infrastructure for a site that could be hosted on a $5/month VPS with even a little bit of tuning and caching. Complexity for the sake of complexity is called stupidity unless you're doing it as part of an educational experience. Going to production with needless complexity is a great way to ruin a good idea.
>Cost is not the same as value.
I guess it's a good thing I didn't state that cost is the same thing as value?
If we’re discussing major corporations, they are willing to pay a premium for predictability and reliability of service. Cloud vendors provide just that: if you want more resources, you can scale up until you hit capacity, then you can call the account manager to increase it. Very little downtime (except for the occasional outages). All of these are known processes.
With operating your own DC, you have to hire really great people and hope they don’t fuck up. You have to plan much in advance for capacity changes. It’s very much possible that something goes wrong in your compute infrastructure and simply nobody in your team has the expertise to fix it. Put it another way: can you pay/hire the same kinds of engineers that work at AWS/GCP? If not, cloud is the safer route. Cost does matter, and this is why companies sign multi year contracts to get discounts. But business continuity is critical.
What does that have to do with the cloud? A company where the "cost of IT is more expensive than the revenue" or has "needless complexity" is making bad decisions and will execute poorly whether it's cloud infrastructure or internal systems.
But if you need a server then (in most companies big and small) it's far easier to spin up a VPS in the cloud than deal with internal IT.
> "I didn't state that cost is the same thing as value?"
You're arguing over cost, but the cloud is often much greater value compared to traditional IT. Spending 2x for 5x more value is a valid decision. If it doesn't work in your situation then don't do it, but it's not the same answer for everyone.
Money is complicated because you have income and expenses. Time to market and velocity can help you get more income. This needs to be balanced by expenses.
When you're growing or your needs are uncertain than a hyperscaler can help you get moving quickly. This aids in time to market. But, you do this at a cost. Hyperscalers cost more than running it all yourself.
If your workloads are steady state or changing at a predictable rate than it can be cost effective to run your own infrastructure.
There are other elements to all of this such as global location of hardware compared compared to end users.
Money is the issue because companies are in business to make money. Sometimes companies take a loss now in order to make money later but it's always about the money.
You have some really good points, but your premise is simply not based in reality.
You're painting the picture as if AWS is the 2nd version of the IBM saying: "No one ever got fired for using AWS". This is simply not true...at least not the part where "IBM" (i.e. AWS now) is somehow inferior to the competition.
> If you are large enough to need to build your own datacenter instead of renting floorspace for pennies on the dollar from Switch or Equinix (although Equinix is kind of pricey) or Windstream or insert provider - then the datacenter is a rounding error in your 10-year IT budget.
Then you clearly don't understand the TCO of using a data center vs cloud. The datacenter cost itself isn't the cost - the people who operate it are.
On a computing resource basis can you operate SQL Server on a rack in Equinix for cheaper than that in AWS? Sure, but who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.
If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.
P.S. - For the record, many large internal IT groups do in fact do simple lift and shifts from traditional data centers and don't fully realize the value of cloud computing resources. However, these companies will usually mature eventually and evolve services to ones that are more advantageous (i.e. setup SQL Server on EC2 and the migrate to RDS). AWS (et al) have the ability to do this with ease.
> On a computing resource basis can you operate SQL Server on a rack in Equinix for cheaper than that in AWS? Sure, but who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.
Who cares? This is IT and is not relevant. And they have Brent and he will solve everything.
Surely for any company of size you understand there will still be people managing the AWS infra. Having tens to hundreds of small dev groups go at it willy nilly is a recipe for disaster and overspending. Someone still has to patch the server instances, manage the people who access it, train them on how to use AWS, etc. Those costs don't go away because you've moved to the cloud.
> Those costs don't go away because you've moved to the cloud.
Sure, but you can go from "I have 5 people managing my 150 instances of SQL Server" to "I have 1 AWS sys admin". The point wasn't that the costs are eliminated, it's that they are baked in.
Generally at big companies you needed people to manage the infrastructure, and people to manage the layer on top of that infrastructure though. With clouds you can consolidate those groups into one group that is smaller.
I think Cloud still costs more overall, never been convinced it actually saves money. I do think it tends to increase developer velocity though, and that can be worth it.
> If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.
This is especially true for more than basic services. If all you need is a bunch of Linux servers, sure, you can find people who will keep them patched for you without too much trouble (although it'll almost certainly turn that “rounding error” into a serious number) but once you start building things which are higher level with legitimately hard problems to solve and need to have good solutions for all of the monitoring, security, ops, etc. needs that staffing becomes harder and you're going to have to sell your executives on the idea that the best use of your expensive A-team is cloning S3, Lambda, etc. when you can get that benefit immediately without a massive up-front payment and have the same people working on a strategic benefit.
I'm trying to be charitable here, can you expand on that and explain why that's not an invalid comparison? For example, the word immediately after your quote was “S3”. That's not something the CNCF can solve for you since one of those is a service and the other is an application you can run — at the very least you need an _extensive_ ops and security if you're relying on it as much as S3 – and, assuming you're talking about Rook, you're comparing it to one of the largest, highest-scaling services in existence but here's what you're on the hook to do if you do it yourself:
* Lower-level storage for whatever you pick for the previous point
Each of those, and the combined service you build, needs staffing for operations, monitoring, security, etc. support. If you follow any security standards or are in a regulated industry, you're on the hook for documenting compliance on all of those, too, especially if you need to be able to demonstrate things like immutability.
You certainly can do this but that's a number of full-time jobs requiring expertise. If you have enough storage usage you can potentially justify the savings over the rates you'd be able to negotiate but that's a big up-front cost which you hope but are not guaranteed to recoup.
I took your previous comment to be about on-prem cloud technology. It exists and does not need to be "cloned". If you were taking about staffing resources, we've pretty well established you're going to need staffing to manage both cloud and on-prem tech.
You need to document whether it's on-prem or in the cloud.
I excluded Minio because that's not branded as a CNCF project but that doesn't make the comparison less invalid: if you're building an S3 replacement, you will need to spend a lot of money up front hiring people, buying hardware, and building an equivalent service. That's a significant 7+ figure commitment just to get to the point of saying that you have an equivalent service.
If you buy storage by the petabyte or use a large enough amount of network egress, you can definitely make that money back but it's a major commitment.
Again, you don't need to build an S3 replacement; you can deploy it onto k8s. This is the same skill set whether in a public or private cloud.
Agree, I don't think anyone is buying SANs that don't need them. If you're spending 7 figures on your AWS bill, you should be looking into other options.
> Again, you don't need to build an S3 replacement; you can deploy it onto k8s. This is the same skill set whether in a public or private cloud.
That is building an S3 replacement. You aren't writing every line of code personally but you are selecting components, configuring them, operating the service, and testing it, which is a major commitment. Simply running k8s at an equivalent level is a non-trivial commitment.
> Agree, I don't think anyone is buying SANs that don't need them. If you're spending 7 figures on your AWS bill, you should be looking into other options.
Note that the 7 figure commitment I mentioned was what you'd need to build an S3 equivalent. Simply having 24x7 staffing and hardware easily puts you in that range.
Nobody is arguing that a startup needs a datacenter; that is a ridiculous straw man. I'm well aware of the costs, having managed multiple colos as well as multiple cloud footprints. I also know you can lease most hardware, if you prefer not to depreciate assets.
You're casting this as a much more complicated concept than it is. It is essentially the same old "build vs buy" argument. No, it does not make sense to manage a datacenter to run a single wordpress blog, nor does it make sense for a telco to run their infra on top of a public cloud. These are all truisms.
My point stands that if you have a seven-figure AWS bill, you already have plenty of associated staffing costs. Once you hit the break-even point between the two, you can decide whether you want to continue to pay linearly, or go down the path of increasing RoI.
You still need the IT team to babysit the AWS infrastructure.
I've seen this "even a lift-and-shift reduces TCO" argument before, but it only ever works if you pretend that AWS lives up to its zero-management hype. Maybe, for someone, it does. Not for any deployment I've seen. This never prevents management from pretending, of course.
EDIT: ah, I've seen that the argument has retreated to "it reduces IT headcount." Again, maybe for someone, but I have yet to see it actually work, and every time management has just hustled the TCO calculation.
> You still need the IT team to babysit the AWS infrastructure. … it only ever works if you pretend that AWS lives up to its zero-management hype.
Who is saying zero-management? If you have a consultant lying to you, that's not a cloud problem and I've heard the same people lie about all kinds of on-premise services as well.
What I've heard the major cloud providers say is true: you need _less_ staff and you can spend time on different things, which has been generally true. For example, instead of needing to configure a bunch of network hardware (which is a distinct skill you need to hire) you can have a general devops person use Terraform to manage routes, firewall rules, etc. without also needing to be a Cisco expert and spending time operating the underlying hardware.
The key question to ask is where your infrastructure was to start with. If you have an A-grade ops team, a lift-and-shift will probably not be much of a cost savings because you presumably already have economies of scale working in your favor, unless you have enough bursty workloads to pay for it that way. The catch is that most places do not have that high-end ops capacity so what you're really doing is taking your limited IT capacity and reclaiming the time which is currently going into things like managing Dell firmware updates.
> who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.
You're implicitly arguing that NoOps is not only a functional model, but also scalable and much less costly than having a {Dev|Sec}Ops staff. The last time this was publicly debated between Adrian Cockcroft and John Allspaw, it was immediately obvious to everyone that there is no such thing as NoOps. You can have developers support their own apps in production on the cloud, but that takes away time from developers and comes with its own set of downsides (most prominently, a disinterest in what they consider mundane or boring). Once your developers are spending, in aggregate, more than 40 hours a week do ops work, you'll gain efficiency by hiring a DevOps Engineer whose entire career focus is all the things developers aren't typically great at. I've seen this in-person as an employee at medium, large, and a Fortune 50 business. I haven't yet seen a single app, not even serverless, that requires zero maintenance. I have also seen highly-distributed, performant apps run on cheap hardware with no SAN or VMware to drive up the cost and complexity. That same app, made up of mostly microservices, was much more expensive to run on on the cloud.
You might want to stop throwing around terms like TCO when you also clearly don't understand how to calculate it.
I'm starting to be of the mindset that us-east-1 is already "Too Big To Fail". I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.
> You're implicitly arguing that NoOps is not only a functional model, but also scalable and much less costly than having a
I'm not. You implied too much.
> You might want to stop throwing around terms like TCO when you also clearly don't understand how to calculate it.
I do. I never said that people were completely eliminated (see one of my child comments). Everything you implied was that "well devs are doing the effort and that doesn't get calculated". That wasn't my argument - you just added that nuance as a retort.
> I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.
I guarantee you they are not. Also, how many data center providers let you simply spin up your whole environment into another region, point n click, and complete it in hours?
Source - work with 100+ companies with PE folks that sit on boards.
I'm speaking from firsthand knowledge. How embarrassing for you to think that because you know some Private Equity folks (generally the worst people at managing technology), that 100% of the rest of the world is doing (or not doing) exactly what they're doing.
You wrote paragraphs about how expensive it is to hire people to do stuff in a datacenter. What is your argument, if it is not that this is more costly than outsourcing it?
This entire debate should be recast the simple "build vs buy" concept that it already is.
So am I. My reference to PE firms was that my anecdote of the few boards I'm on is supplemented by their experience.
> This entire debate should be recast the simple "build vs buy" concept that it already is.
Because that's entirely the premise of my argument. Did you read the parent comment and what I was responding to?
PS - Say what you want about PE folk, but they are the most ruthless capitalists. So IMHO they are far more incentivized to drive cost efficiencies than anyone else because profit = EBITDA = valuations.
> I'm starting to be of the mindset that us-east-1 is already "Too Big To Fail". I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.
If you're not blasting your AWS account manager and asking what their plans are to avoid this happening in the future, then your CTO is not doing their job.
If AWS itself is not internally working out how to "get off us-east-1" for its global services, then it is failing in its role as infra provider.
AWS has a lot of stuff about how to make your infra redundant and resilient to failure, in their environment, mostly about replicating across AZs and then across regions.
The fact that their "global" services are not actually "global" but are "everything goes to us-east-1" is broken and if they're not fixing it, then that is when you start looking at other alternatives.
>Then you clearly don't understand the TCO of using a data center vs cloud. The datacenter cost itself isn't the cost - the people who operate it are.
I guess I find that a bit humorous. I've been architecting solutions that span on-premises and cloud for nearly a decade so I'm intimately familiar with both. I don't actually have a horse in the race. You want cloud, buy cloud, you want on-prem, buy on-prem. But the marketing about one-size-fits-all is EXTREMELY tiresome. And the idea that every single business can or should be moving their entire infrastructure to the cloud is as well.
The fact you imply that somehow you don't need people to manage the cloud makes me seriously question what at-scale deployments you've been a part of. I have yet to see a single enterprise customer save money on staff by moving to the cloud. More often than not what they end up doing is laying off the "tape jockey" and replacing him with an SRE who costs 3x the salary. Congratulations? Now you're paying more for an SRE, and you're backing up to S3 at about 5x the cost of what you were spending on tape? But I guess the 7 years of backups you're legally obligated to keep are more agile?
>If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.
I don't think the concept of cloud computing is lift and shift and never said it was. But that's not really relevant to OP's question, what is relevant is what's happening on the ground.
What's actually happening is that most established businesses "moving to the cloud" are lifting and shifting and finding out the hard way. I'm not saying that's what they should do, I'm not saying that's at all a good idea, I'm telling you that is what is happening. And if you think "well they'll mature into a more cloud friendly model" - you're as delusional as their management. What ACTUALLY happens is after they lift and shift and get a year's worth of those bills, they lift and shift the stuff that never should've been in the cloud in the first place back on-premises. Hopefully by that point they've learned their lesson and have a thoughtful evaluation of their applications: re-home, re-write, leave in place.
> But the marketing about one-size-fits-all is EXTREMELY tiresome. And the idea that every single business can or should be moving their entire infrastructure to the cloud is as well.
100% agree. Didn't mean to imply this. As with anything....it depends.
Cloud providers (especially Amazon) can definitely be a bit greedy / shady about pricing and lock-in, but the key thing is that they have room to lower prices whereas a bespoke data center does not.
Most data center machines run at a tiny fraction of their capabilities; seeing a CPU running at 25% is definitely something that stands out. Because of this, cloud operators can oversubscribe these machines in various ways, packing workloads of multiple tenants onto a single box while a single-tenant bespoke DC can't. They can also do things that schedule DC-wide resources (namely power) in ways that renting space in a colo cannot; they can shed load in insightful ways when a power overtopping event happens that can minimize the impact (e.g. throttle / nuke VMs that can operate in a degraded mode before impacting other critical services). No hetergeneous, multiple-private tenant DC is going to oversubscribe their rack- or row-level power delivery to tenants and require them to hook in to a smart system to manage overtopping events; they just won't trust their tenants to do the right thing, as they have no power over the tenants' machines and power draws; the cloud providers do.
At the end of the day, the economies of scale are just hard to beat, in terms of COGS.
The real question is whether they'll pass the savings from this on to the customers or the shareholders' dividends, and so far it seems like it is the latter.
Wholehartedly disagree. I've been in large scale IT for 20+ years, working for deep pocketed companies. The costs of running old school IT with on prem data centers were orders of magnitude higher vs. what we pay for well architected serverless solutions these days. I remember having a $30MM/year line item for Oracle alone, running on expensive hardware that was sitting idle 90% of the time. Time sharing expensive hardware is the way to go.
Another thing is costs of building something new. In the old world it would mean doing the math on resource needs now and in the future, since adding Hw meant capital expenditure. No such problem now and its easier to get started, see if it works, and scale or kill early.
And of course as others have mentioned, keeping it all running requires expensive expertise. Esp. now with the internet that looks like automated wild west. You cant release a service without it being auto-probed and auto-attacked within 2-3hrs.
Nope. With the cloud, the costs have went down for any large coMpany, even those that just moved the old crap to VMs running on someone elses hw.
There is, but there's less if you're using managed services where possible and it's visible in a way which wasn't true for many large organizations. I can go into any cloud provider's console and see a detailed breakdown of what I'm paying and why, and how much of it is over/under-utilized, but getting that same report from an enterprise IT department is usually a project.
The problem is not just the datacenter itself but 1. all the personnel needed to operate the datacenter/private cloud 2. the lack of agility when everything needs to go through internal IT provisioning (at some companies this is a huge bottleneck).
Fantasy football is actually interesting in that it happens “off hours”. So if you are a company that has an “on hours” business, running Fantasy Football is free if you utilize multi tenant hardware.
> Fantasy football is actually interesting in that it happens “off hours”.
I entirely associate fantasy football with working hours. Ditto a variety of other kinds of sports betting. Like, I know people do these at home, too, but there's a lot of it happening during business hours, at offices.
It's all good until you remember you have to now higher an entire team to manage your bare metal servers. Companies don't like adding new employees, even when it's comparable in cost to hiring an outside vendor or paying for an outside service. Firing 10 people is very messy, reducing your AWS budget by 20%, however is not.
Are you also factoring in that you often need 2+ DCs for redundancy? That seems like a lot of operational difficulty unless your company already had a physical presence in multiple locations.
I know there are also a bunch of requirements for where you can store user data these days due to privacy regulations.
Exactly. The reasons to adopt (or not adopt) cloud, like any technology, should revolve around business questions, not technical ones. Of course, technical ones are critical to business success, so you can't ignore them, but they are subservient to the business concerns.
Security is complex but many companies find it easier to offload it to managed services and best-practices/reference architectures in the cloud compared to janky internal systems.
Offload what exactly? You mean making the same decisions, but through an opaque badly documented panel? Or you mean paying somebody to keep your OS up to date?
Going into the cloud does not alleviate any security decision. You can't buy security.
Of course you can. In this case, many of those decisions are already made for you in the cloud.
A simple example would be using RDS for a database service. It's secure by default and only the credentials are an issue, compared to running a database of your own somewhere internally that may be exposed in dozens of ways. Less decisions, less surface area, more documented reference setups to follow, more scanning and analysis tools means better overall security posture with less work.
On a previous job we had a huge GPU server on prem because we wanted to use it for deep learning.
It would have been much, much cheaper to rent gpu in the cloud and when we knew _something_ about our workload we could buy an on prem and maybe rent extra if needed.
But we weren’t cleared for cloud.
Regarding in-house expertise: I think most companies would like to have in house expertise but they can’t hire the expertise they need so they use consultants.
It all comes down to utilization as a percentage of time. If the workloads are infrequent enough, GPU cloud providers on going to crush the onsite equipment.
However... I was also sort of surprised, because last time I priced this out, you can have pretty low utilization rates of the equipment and still come out ahead.
the other thing to consider though is that if you are under pressure by management to not utilize GPU cloud, you might do fewer experiments than if you had at least a GPU workstation under the desk. (that, presumably, you can share with other workmates)
I've looked for some GPU on the web and it's super expensive to rent. I was expecting to rent that for say 50$ per month (considering I can rent some webspace for less then 10$ a month) but everything was more expensive like 600$/month (I look for entry level GPU, used just a bit, I'm a student) ... Did I look correctly ?
if you want an always-on gpu for $50/mo? That's unreasonable. Consider how much "building from scratch" even a cheap "minimal" GPU unit costs (let's say 3k GPU + 1k supporting PC), whereas a cheap "building from scratch minimal webspace" is a fraction of a single CPU unit, so let's say $100-ish/unit dollars, to be really generous. A rented non-fractional GPU should be easily minimally 40x the cost of a website.
Don't expect these ratios to be authoritative; there are obviously different economies of scale and many many details I papered over, but this should just give you an idea of the back of the envelope "relative costs"; just that 5x is way out of the range you should have guessed.
I appreciate your point about "they can't hire the expertise they need" but I feel it's missing the "because": because those hires are so expensive? Because they only exist as consultants? Why can't they hire the expertise?
because nobody wants to be hired for the position. it isn't even about the money, there's simply no candidates - and any candidates competent enough are already consultants and/or freelancers.
For company I am working at, it is acceptable to pay more for even long term consultanst, than to pay market price for experienced developers. The reason explained was, that it is different budget line - HR does not want to "overpay" for emplyees, so consultancy is the only option.
I would so that about 3/4 places I've consulted, I wouldn't want to work full time.
Mostly because they simply didn't know how to run an IT office, and so they couldn't build a culture where good IT people want to work. When they finally get a strong candidate, they end up overloading the poor hire, because finally, it looks like they can get their infrastructure fixed or dream project off the ground.
Like anything it's just pros and cons. Sure you save money by doing on-prem or just buying servers and colocating, but you also get all the headaches that come with that.
I used to work at one of the most popular fast food chains. I had one project that was internal to them and one that was customer facing. The internal product, a search engine for documentation ran on-prem while the e-commerce store ran on AWS. Running the former on-prem was just fine.
I'm sure glad we were on AWS for the customer-facing site, though. We'd run ads during the Macy's day parade, college national championship, etc... dear lord I could not imagine wrangling the necessary people to scale that out from an infrastructure perspective. It was very nice just doubling the number on our ASG, failing over to the secondary database, launching another DB instance with double the capacity, and repeating for the second one.
Between those popular ads and DDoS attacks I'd say AWS was a win there, but internal tools don't benefit from the cloud as much. Also not an MBA, but I feel like that's a rule a thumb you can add to the repertoire.
or, more likely, it would have been impossible to scale at a short notice and the site would slow down (with luck) or go offline for a bit (most probable).
It seems no company wants to be in the business of in house IT Systems. They think moving all their infra into the Cloud absolves them of all the responsibility of owning it themselves, they don't give a shit how much it costs but then tell you everyone has got to save money and tighten their belts.
MBA here. The cloud is not a good value, the cloud handicaps your organization by placing your most valuable resource - how you compute - in 3rd party hands. Doing the financial math, the cloud is absurd, it is a very bad value; if it were a better value, the cloud providers would not have such high profits. Doing the flexibility math, having one's own datacenter - even a rented colo cabinet - provides far more flexibility than working through a cloud provider's ticketing system. Doing the organization knowledge math, having in-house skills to create, modify, and completely customize one's backend infrastructure with the knowledge that you're numbers are pure - nothing but your jobs are running on your hardware - grants you the visibility to optimize effectively.
I've done this: built a 17 server cluster, housed in a colo, for a high compute startup producing 3D avatars from photos automatically. The hardware for the cluster, including a Federal Reserve class hardware firewall, was just over $50K, with a month to month colo bill of $600. This was in contrast to $96K per month if the same configuration were at AWS. This was 7 years ago, so adjust the prices, but you get the exponential higher expense point.
On the other hand, what if your tiny company spends only four figures per month on AWS and has only a single developer who also acts as a sysadmin as needed? Is it even practical for a company operating at that scale to colocate servers in an environment that has anything like the private network and multiple per-region availability zones of AWS? Not to mention that, presumably, the developer would have to implement and test their own equivalent of AWS's auto scaling and multi-AZ database services. To me, AWS is worth the money.
We colo’d our own servers as a two person team. We didn’t have multiple availability zones, but we’d have the ability to fail over to leased machines if we needed to via automated builds and frequent offsite backups, and that was good enough for our needs. But we never needed to over the few years we ran this setup. This was after running on leased dedicated machines (which is a good first step so you can figure out what you actually need without putting out the money for hardware).
A private network is ridiculously easy to set up. In our case, we had two switches, one public, one private. Every machine had 4 NICs. Just bind some to public addresses, some to private, plug in to appropriate switch.
Unless your traffic is extremely bursty, auto scaling is probably more trouble than it’s worth. We were serving millions of users with three dual-socket machines. And if you really need it, you can try to handle base load with dedicated servers, and bursts with AWS infra. But at <$10k/mo on AWS, you’re not getting anywhere near the horsepower I’m talking about.
That $600 figure doesn't include the minimum in-house 5 person staff + 1 manager necessary to run a 24x7 high-9s operation that could run operations such as live-migrating workloads to standby systems in order to fix faulty hardware. Additionally that same $96k could be used to run your compute in multiple datacenters. Back of the napkin math suggests that it'd be a wash once you expand to only three DCs.
I understand it was a startup and may not have needed high-uptime ops from day 1, but a lot of organizations do, and the cost is not that dissimilar.
Yes, it was a startup, with a staff of two. Once we had the cluster up, self monitoring, and rotating on command it is literally a half day a week to maintain the entire setup. The hardware firewall was our best investment, as it load balanced while fending off significant DDoS attacks from hackers. It also is entirely required to use an experienced network engineer for such a task. Far too many people try to wing this, and that's where all the publicized horror stories arise.
> I've done this: built a 17 server cluster, housed in a colo, for a high compute startup producing 3D avatars from photos automatically. The hardware for the cluster, including a Federal Reserve class hardware firewall, was just over $50K, with a month to month colo bill of $600. This was in contrast to $96K per month if the same configuration were at AWS.
You also have to include the human time spent setting up and maintaining those servers. And also the fact that you can probably use a managed service on request instead of 17 servers running constantly. Not even mentioning redundancy.
Yes, the human time is an investment that pays back multiple times over. Why the core of a technology business would be in the hands of a 3rd party dismays me.
Of course, those 17 servers composed dev, staging, and live, plus fallbacks, ultimately composing quite a few APIs and related services. They were all used for high compute, where a task would floor 32 cores for 3.5 seconds, and that was essentially what the cluster was for, beyond the accounting and tracking of its use.
Most businesses are not technology businesses, they deliver goods or services to their customers where IT is part of their costs to deliver.
Investing in IT people to run that bare metal is a huge expense that doesn't improve your goods or services directly.
As for "where a task would floor 32 cores for 3.5 seconds"... means that your internal IT infra had to support that maximum requirement headroom 100% of the time, even though it may not be used for 99% of the time (depending on load patterns etc).
Not sure how an MBA helps here, but if all you are doing is straight compute, then yes, cloud is a waste of money. The benefit of cloud comes when you have a bunch of different teams with different optimal solutions such as nosql, sql, queue services, data processing pipelines, data warehouses, object storage, CDNs...
Often it's an OpEx vs CapEx play. Also large orgs crave treating staff as interchangeable parts and IT hiring is thorny. Throwing too much money at the problem can be preferable.
Enterprises would rather not maintain inhouse infrastructure for many of the same reasons they engage MSPs despite persistently shoddy services.
It depends on who's running your company. The bog-standard MBA's in middle management love to minimize capital outlay, preferring to expense everything possible. Sure, there are tax advantages, but it becomes a rule-of-thumb in most Fortune 500's, and winds up infecting IT as well. What a lot of blue chip companies STILL haven't grasped is that IT is as strategic, differentiating, and important as any custom machine they can design, build, and install on an assembly line, and the software engineers, as much or more, important as the product engineers.
Big corp financing is weird, and IT is often its own cost center responsible for its own budgets. This means that every time my department asks IT to do something for us they send us a bill. And that bill is always several times more than going to an external provider. At a place I worked, hosting in-house would for example cost a minimum $150/server/month for something with the same spec/performance that would cost $20/month on AWS. It could also take up to 10 days to get that (virtual) server set up. As a project owner and someone responsible for my project/department budget using an external cloud over in-house hosting makes rational, economic sense
Building up an IT infrastructure that can provide services at roughly the same quality and cost as AWS takes a large investment in talent and infrastructure. And many companies don't see the value in making that huge upfront investment for something they might not need. Better to start with AWS, and once you get to the point where you can lower your costs by 50-70%, then start looking at moving in house.
> Building up an IT infrastructure that can provide services at roughly the same quality and cost as AWS takes a large investment in talent and infrastructure
no, you missed an important point here.. no one, at any scale, has been able to compete directly with AWS on the combination of service, scale and cost per unit.. its intensely competitive
You have to run the numbers for your specific company, expected scaling behavior, compute work load, and expected system administration overhead and compare all scenarios.
I will say that if your work load is relatively fixed in magnitude and especially if it's bandwidth intensive, you can probably beat the "big cloud" providers by at least an order of magnitude. If it's extremely bandwidth intensive you should never consider mainstream cloud as outbound bandwidth is absurdly overpriced... like 1000% markup or even more. I can't exaggerate how insane bandwidth pricing is.
If you are just prototyping something, your work load is highly elastic, or your product is rapidly evolving, the flexibility and outsourced sysadmin of cloud will probably make sense.
Edit:
There are edge cases too. If your work load is, say, inbound bandwidth and storage intensive, AWS might be great since S3 is cheap and inbound bandwidth is free. Just keep in mind that you might pay a lot for egress if you ever need that data out. Will you?
Almost all “on prem” business IT is managed by external contractors. I was one of them. We served about a third of the commercial premises in a small Midwestern city, driving out to visit the server closet, change backup tapes, click through Windows Updates, and perhaps make the rounds of desktop users with any questions or concerns. We sold a monthly package of so many labor hours, and we’d visit each customer site about once a month. We’d get the keys from an office administrator who perhaps knew how to provision user accounts. The biggest, richest customers might have had 1-2 help desk on staff, probably not domain admins and certainly not empowered to do any decision making or design. That was for us.
It was quaint, adorable, personal, and hilariously inefficient.
You're talking "Mom and Pop" scale businesses that have minimal computing needs other than printing and Internet access. Most "on-prem" IT is much larger. My company (also in the Midwest) has mainframes, SANs, vSphere deployments, Nutanix, as well as supporting ~3K users.
These businesses ranged from "Mom and Pop" to about 50 employees. There'd be a domain controller, Exchange server, file server, PBX, SQL server, and application servers for 2-3 line of business apps.
It seems very clear to me that these kinds of businesses are much better served by cloud. No one of them is that valuable of a customer, but there are so many.
Companies that are large enough to have an IT department with multiple highly skilled professionals onsite, or who could even contemplate doing a real data center (most "on prem" is in a closet), I agree is a different calculation.
I have worked on reliability for some private OpenStack deployments, on-prep K8S deployments. Views are my own and don’t reflect on any open source community or company.
To operate a production cloud you need expertise at all parts of the stack. This includes network, kernel, maybe firmware, orchestration, etc. You also need adequate planning where your on-prem data center teams continuously upgrade your hardware/firmware and add capacity. From an organizational perspective this is an entire VP or SVP level organization depending on where you are.
Operating large infrastructure is hard. If it’s not your core competency there is really no point in dealing with it. Either way, it takes enormous effort to hire and keep a team skilled to operate infrastructure. You will lose your team to big-tech eventually.
You don’t need external consultants to manage your cloud infrastructure. Your existing engineers can skill up to use it. Public cloud will commoditize what typically takes a lot of time to plan for capacity, DNS, networking, scaling, etc.
In short - its prioritizing speed of execution over everything else. The opaque billing is just cost of doing business. But unless tech is your core competency, opaque billing is much easier to deal with than all the other situations that come up with managing and maintaining your own infrastructure.
I bet in the 1920s, people were questioning whether or not businesses should generate their own electricity on site or whether to go with the new-fangled "cloud" solution.
The question for businesses comes down to economics, of capital expenditure vs operating expenditure. If I buy my own hardware, I have to spend capital and depreciate it as part of the operating expenses. If I lease hardware (whether on prem or cloud), then that's pure operating expense.
There are accounting and tax implications. There's also the "stick to your knitting" of a business. Do I need my own payroll service, or do I outsource it. Do I need my own accounting, or do I outsource it. Do I need my own cleaning service...
> Problem: Managing server hardware takes money and people.
For most, this was never the effective reason.
Most rationales behind cloud migrations are likely based on quick scaling, which is meaningless in most cases - except for the signalling of committing to a future where demand hockey-sticks upwards.
It's valid if you are netflix, but how many netflixes exist in the world?
> It seems to be a universal truth that computing people try to solve all problems with another level of indirection.
This isn’t about computing but about specialization. Should McDonalds be in the business of operating Data Centers? Or should they just buy it from those who do it best and focus on selling food?
> Should McDonalds be in the business of operating Data Centers? Or should they just buy it from those who do it best and focus on selling food?
Should they be in the business of cleaning bathrooms, or should they just outsource that to a temp agency? Sometimes specialization pays, and sometimes it doesn't. Perhaps it doesn't make sense for McDonalds to run its own web and corporate servers, but doing so might make a lot of sense for more tech-centered companies. And I'd guess a server farm is miniscule compared to the size and complexity of their ingredient supply chains.
Cleaning bathrooms is not the same level of complexity as operating datacenters… you can train a janitor pretty quickly; training a software engineer is much more involved. And even with trained software engineers, operating a reliable data center is just hard.
> Mr Garman counters that the higher price of moving data off a cloud (“egress” in the jargon) reflects the higher costs of that exercise. Almost by definition, customers leave with more data than they entered with.
Uh, they charge more per byte, that's not what's going on. I don't know if that was AWS's Garman being deliberately misleading, or the Economist journalist misunderstanding.
>Businesses’ main motive for moving to the cloud was never about cost but “scalability”: having access to additional computing resources with a few clicks.
That's not accurate because many businesses do migrate for cost reasons. Especially for internal non-public facing servers like email. E.g. They moved from self-hosted Microsoft Exchange servers to cloud-managed email like Office 365 Exchange Online -- or switch to another cloud competitor like G Suite.
>Ms Wang and Mr Casado have suggested that firms should think about building their own private clouds to keep costs down. So far few firms have opted for such “repatriation”, which is both costly and makes it harder for businesses to enjoy the benefits of essentially unlimited computing resources in the public cloud.
If the company's business profit is not in the tech sector (like The Guardian), this means its IT datacenter operations is not its core competency which makes it hard to build internal IT that can keep up with AWS capabilities. Tech businesses like Dropbox, Facebook, can hire & maintain skilled IT staff like AWS but most other non-tech businesses can't.
They say the "main motive", not the sole motive. Cost is also a motive, for sure, but do you consider they'd move to reduce costs if there wasn't scalability?
> The Guardian newspaper tried building their own private cloud and failed
What I see often is that it's hard and costly to replicate the whole cloud experience in-house. OpenStack is a behemoth to operate. However, if you build something more specific to your needs (and often mix in selected public cloud services), you'll get a much cheaper stack.
Small businesses can often run very efficiently with a few rented physical servers in a colo, G Suite and AWS S3.
Larger ones can provide a platform specific to the technologies used in-house and combine with public cloud for any other tech stacks.
Why are you recommending S3 when Ceph exists? You almost literally just throw hard drives at it and it gives you remote block devices for VMs, a NFS style shared FS, and an S3 client compatible object store.
People always think Ceph is super complicated but I promise you it isn't. I worked at a place where my (3 person) team and I operated a 1 PB Ceph cluster replicated across two datacenters along with literally every other service needed to run a production web app. It was one of the lowest maintenance services we deployed -- Kafka and our RabbitMQ clusters were the things that always gave us trouble.
I was on a team that had a tiny (a few GBs) Ceph cluster because it was the easiest option for high availability of these files that we really wanted to have on premise. There were never any issues so I can believe what you're saying. Still, for the amount of data I work with, the cloud storage usually wins.
Others may have other preferences. This is what worked for me. I just find S3 quite cheap for storing a few (tens of) terabytes plus some glacier archives. For data I often want multiple locations redundancy and encryption. That gets more complicated than what I can save.
I guess the equation changes when you have more data or looser requirements. That's actually my point. It's very hard to recreate the whole AWS experience, but it's quite possible to pick any piece and do it much cheaper.
It's basically consultingware for IBM/Redhat. It tries to be everything you could ever possibly need, the interactions between the components are brittle, and it has all the complexity of a huge k8s deployment except your ability to affect what's going on underneath is like trying to control an angry beehive with a 20 ft pole.
Think Urbit but for sysadmins and you pretty much get the vibe.
Ever tried installing it? It's an incredibly complex piece of work with boatloads of services to configure and debug... and upgrade.
At the time I last tried it (some two or three years ago):
- most of the documentation was written assuming an rpm-based distro and not Ubuntu or Debian (I remember especially having had trouble with subtle differences in iptables/nftables between the distributions)
- on top of that it was right in the middle of a python2 => 3 migration with the result of shit breaking left and right
- it was/still is written in Python which means a second or more for every CLI command to initialize, much less actually do something
- half the instructions in the documentation seem to never have been actually tested, which combined with the two points above made for a really punishing "experience", especially with more rarely-used features such as k8s-in-openstack
- the documentation itself still is (just looked it up) badly organized, with you needing to read four guides (Install, User, Configuration, Ops/Administration) per OpenStack component. Ideally, there should only be one guide for Installation/Upgrade (including Configuration and everything from Ops/Admin) that has everything in it to get all supported parts of the component up and running, one User's Guide that shows how the component is used and what the best practices are, and one Troubleshooting guide.
If you can set it up into a working configuration, it's a nice project for sure. But the road to that point is rough, and you will likely not even want to think about an upgrade simply because of how complex it is.
I worked at a place that had a large, underutilized data center. The DevOps team was stuck in the past and operated too slowly to provision capacity when it was needed so a lot of it was just fallow. They had spent months and months working with a vendor to get these into a k8s fleet, but I never saw the outcome. The devs teams in my office were just throwing shit up on Azure or even Heroku and just not asking permission. We got substantial increases in revenue and the incremental cost of hosting was negligible.
I've been there as well (modulo cloud brand to which we moved). I understand this reason for moving to cloud but it makes me sad.
It worked like this - you had to fill a form to request a VM. After a few emails (and days because time zones), you got a few years old RHEL VM without root access, but also without any real support from the ops organization. They tried to provide the generic cloud experience with VMs and failed.
Most companies that I see operating private clouds are using Kubernetes. It's much more flexible than OpenStack. It also has the advantage that apps are more or less portable across private/public cloud infrastructure.
The comments here are largely unquantified opinions that haven't really changed in 10 years or so, more that there are far more people that have those opinions.
Ten years ago I loved the cloud because it broke the monopoly / organizational dictatorship of IT in large organizations, always a wellspring of "daily wtfs".
But now the cloud is becoming too powerful, and too small in number. We NEED a competitive-ish on-prems approach... but... what is it? Does anyone have a template to do so? They aren't well advertised that's for sure.
AWS is NOT CHEAP. It may be slightly cheaper than the old monolith IT orgs' DCs, and far more flexible, but it's not cheap. There has to be a way to accept less flexibility for substantial savings over reserved instances and negotiated discounts.
For christsakes, a Raspberry Pi would make a perfectly good server for massive ranges of use cases. 20$ and ultra low power.
The reason there are many unattractive alternatives that are underused is the fact that they are unattractive. Nothing has probably changed in years in the traditional server renting space.
Why are you comparing it to selfhosted? BareMetal has been a thing for decades and outperforms the "cloud" on cost and performance for the vast majority of usecases.
Except that you have high initial costs in setting up this hardware, especially in a scalable fashion; it quickly adds up to tens of thousands, plus you need employees that can manage something like that, including 24/7 support or monitoring.
Or at least, that's part of AWS' sales pitch. But it makes sense, if you're just an application developer, you can fire up a hosted, scalable, redundant cluster fairly easily and quickly, whereas if you go self-hosted you need to buy hardware, colocation, and either know or hire people that can set it up and mange it.
Sure, but you need to put it into Powerpoint and sell it as the next big thing in cloud computing: your cloud provider no longer offers you shared, cheap virtual servers that anyone might be using at the same time, but 100% dedicated servers in the cloud, with unmatched reliability, stability and performance.
You just might impress top brass and convince them to get on this bandwagon before they are disrupted.
How is Hetzner not a cloud computing business? You click a few buttons and get a a ready-made infra from an already provisioned pool of hardware. This is by definition cloud. Maybe it doesn't have all the bells and whistles from the big 3 cloud vendors but Hetzner is very much Cloud computing
The problem is social and bureaucratic. Business unit X wants to buy software that solves a certain business problem. Oops you missed this annual budget cycle that is tied to hardware purchasing, better try again next year. Or, great we'll help you buy that hardware, but there is a 3-6 month leadtime. So many organization had an adversarial relationship to their IT group. Even when they didn't want it, the structures prevented solid partnership. On the other hand, SaaS, or self-service cloud (either on-prem openstack stuff, or real cloud aws/gcp/azure) allows groups to manage their own resources while still having a central coordinating function.
Working in enterprise software, I was am always surprised when I find a killer feature may not the functional value of something we've built, but that fact it works around some internal disfunction of operations between internal teams.
That's the opposite side of Conway's law, but the real point is large organizations can be dysfunctional. Cloud computing lets customers bypass their organization's particular dysfunction.
It's TCO through the lens of "do you really want to have to have specialists in that?" Akin to running your own power plant -- MAYBE you can do so "more cheaply" than grid power through a one dimensional cost of inputs lens but all of a sudden you're running a power plant which is a bummer when your customer complains that the burger doesn't taste right
Arguably, the guardian is a tech business at this point. However fundamentally - hosting is not its core competency. If you've ever tried to pinpoint a businesses's core competency the simple test is "is this something that if we fail at, we'll keep trying". There is a big execution risk in building a private cloud, you end up paying this risk both on whether you have the right people to build, maintain, and grow a datacenter - as well as whether you have the right set of APIs to support growing your application.
Most businesses will look at the relative costs and opt for a private cloud solution, then price in the risk and go for the cloud solution. There is no reason the Guardian must control its own hardware.
> There is no reason the Guardian must control its own hardware.
No, but it has potential political ramifications, right?
All the Cloud providers are US. Does this mean the guardian will be more wary about saying anything negative about the US?
All the cloud providers are big companies, does this mean that the Guardian might be less critical of big companies?
Obviously being completely vendor agnostic is difficult; but we shouldn't pretend that there could not be backlash in unexpected ways- even if "hosting is not a core competency".
A thing that comes to my mind is Edward Snowden's revaluations: given the things he talked about (NSA was even tapping Google lines, for example): how could any whistleblower expect to not be exposed by accident if everyone is using a cloud provider in the US?
It just strikes me as marginally naive to assume that nothing can/could go wrong here, you hand off an immeasurably large amount of control to an ambivalent entity and whatever regime they happen to be under.
You bring up a good point. Last year when Parlor was kicked off of AWS, it was a dude awakening. Before that, I had (correctly) thought that AWS doesn't read the data you store and (incorrectly) AWS doesn't care what you store.
I now realize the cloud providers are willing and able to express political opinions and delist people/companies that disagree with political parties.
It very much depends on the business. We offer managed data warehouses, which (a) are major consumers of compute/storage/networking and (b) often have stringent requirements around security and control of data.
We're betting that not everyone will be in the main public clouds for these reasons and prepared for the pendulum to swing back for some applications. Financial services companies for example still have major on-prem / self-managed footprints. They are very sophisticated consumers.
(My company Altinity was the one cited at the head of the article.)
The fundamental promise of the cloud was that it was going to be cheaper over time. That was 100% how it was sold back in the days. “We buy all the hardware to a fraction of the cost comparing to any private company”, and so on. Scale arguments. And “We manage the hardware and the platform so you can focus on features”. Today, Azure for instance motivates the high prices per request with things like security and compliance. I guess it is the same with other providers.
> The situation is analogous to a century ago, when electricity became an essential input
You can do nearly everything "the computing cloud" offers without it : just use servers, install your services on it, manage it, invest in more servers/ more powerfull servers if needed... Cloud computing is not a brand new invention, it's just delegating the server and services management to others...
Whereas, a century ago, electricity was actually an essential input.
> Companies used to run their own electrical generators.
... and have the staff with experience and resources to keep them running. These days, UPS maintenance is sourced out to the cheapest bidder and testing is infrequent, with the predictable result of shit hitting the fan at the exactly worst time.
My first job was IT support for on-prem systems and, as I battle my containerized deployment scripts this morning, really miss those days. Probably just nostalgia but I had named them all after superheroes and it was just way more fun when you got to spec a server and install it, fix them (physically) when they had issues, and panic when the AC broke.
Edit: Also I was way more fit from climbing around a building all day cabling stuff. Genuine question - does anyone know what sort of IT jobs require physical work? I miss those days a lot!
> does anyone know what sort of IT jobs require physical work?
I have seen a few positions in the US for datacenter technicians. Those positions had explicit requirements(probably because of ADA) to be able to lift over X lbs as part of day to day activities.
Hetzner would probably be the perfect working space for you. They build all of their servers by hand on custom racks. You can move around in 14+ buildings in Falkenstein, manage cables, fix servers, work on their cloud offering, ...
COVID has probably changed this, but networking jobs tend to require a lot of racking and cabling work, especially if you’re fitting out a new work area. Even with the majority of your infrastructure moved to the cloud, you’ll still need APs or Ethernet jacks wired into a LAN for the end users.
I did IT support for a federal agency headquartered between two buildings with multiple floors and was on my feet all day. About once a week I’d actually have to run as someone was experiencing a time-critical hardware issue with only minutes to spare, heh. I kind of miss it. But the pay is nowhere close to what I make as a SWE.
Due to how the difficulty system works, the real-world price of the reward for mining will always be equivalent to the cheapest energy price in the world. If the price suddenly increases, more miners get in, which pushes the difficulty up and now you have to expend more energy for the same reward - similarly if the price drops, miners who can no longer make a profit go offline, lowering the difficulty - in the end it's an equilibrium.
I don't think it was ever profitable unless your energy prices can compete with what industrial-scale miners get in favourable locations (right next to a power plant in an area with too much surplus power, etc).
What Nextgrid is saying is a common, incorrect stereotype.
"Is it really profitable to get in, like now?" Yes, but... Long version: Yes, you will make your money back within 2 years almost guaranteed on a new mining rig. There is a minimum cost to get in. For example, at a minimum, you will need a good card at 3k+, which puts you at 3.5-4k just for price of entry. However, you will make that back within a year.
Now the heavy BUT:
1. No one knows what will happen once ETH switches from mining to proof of stake in 2022. Mining profits may go to 0.
2. If BTC price goes to hell, so do your profits in the long run. This is price-risk, regulatory-risk, etc.
3. There are heavy tax implications and a lot of management. For example, you won't get the returns mentioned if you don't properly account for equipment costs.
4. Once you want to make more than $10 a day, there are heavy, heavy set up costs - cooling, proper power supply, etc.
Finally, profitability of mining is back because China is actively banning crypto mining, so other areas can compete now.
"I don't think it was ever profitable unless your energy prices can compete with what industrial-scale miners get in favourable locations (right next to a power plant in an area with too much surplus power, etc"
This is simply not true Nextgrid, see https://whattomine.com/ for roughly-valid numbers. Note that BTC is not anywhere on the GPU list - you do need extremely expensive ASICs to mine BTC. You also need to order them from China, pay a 25% import tax, and overall risk quite a bit.
Final note - if you could get GPUs at cost or MSRP, you would be in pure profit in less than a year. The problem is that a $2000 GPU is marked up to about $3500 atm.
cool. the best answer i got in a long time. here is what i have in terms of pricing.
https://whattomine.com
says rx 6700xt is 48mh/s while 3060ti is 58. this is fine but the pricing is 6700xt at around $680 and 3060ti around $1000. now, 3080 is $1500 but it also has 91 mh/s.
anyways, is $15/Mh a good buy because that is what it seems to be
Not really. On other side news like 'my AWS account racked 40000$ while I was asleep (because of the crypto-miners)' make to the front page way to often.
One way to reduce compute costs significantly is by using a much lighter stack. Typical deployments to the cloud are based on the traditional OS model, and more often than not are a container-in-a-VM solution to deploy some service. The reality here is that there is so much unused, unnecessary and wasted compute resources. Why deploy a full OS and kernel which will only ever serve static web content, for example. And then there's the attack surface that comes with a full OS, running additional services like SSH which may only ever be used once.
Major vendors like Amazon and Google are happy to provide this to anyone and with simple on-boarding, but, as is clear in the article, it's expensive. With Unikraft[0], you can completely re-package your application as a unikernel, automatically. It's a single VM image and does nothing more than just running the application. It boots straight to the main application, it doesn't have a shell, and has nothing else running along side it. The result is a huge cost saving (up to 50%), since you're no longer having to pay for these extra resources just to run the a full traditional OS image plus your application.
Whoops :) Yes I am affiliated, I can't update my comment but I'll make sure to indicate this if/when I write a comment which discusses anything about Unikraft.
It's really cool work the folks at CloudFlare are working on. The trade-off there though is that it's still a process isolated by the OS and so relies on OS-level security via the use of syscalls. Even if context-switching is reduced with Deno, making privileged calls to, say for instance `open()` or `socket()`, still impose wasted CPU cycles as the OS checks whether the process is allowed to perform this action. With unikernels, the call is allowed (and no extra CPU cycles / instructions are called) since the unikernel is its own VM and is isolated by physical hardware separation.
We are launching a pricing page in the next month as we are gearing up for launch :)
One example of recieving a 50% cost saving is from Amazon Machine Images (AMIs). We compared the cheapest official NGINX AMI available on the Amazon Marketplace[0] and then also ran an NGINX+Unikraft AMI. We ran the same workload using wrk[1] and then checked the bill at the end of the month, roughly $80 vs $40.
Most of the costs for using the NGINX AMI + t2.nano come from software (0.29/hr), not EC2 (0.006/hr).
That's like the worst instance you could pick to prove anything about the performance of your platform. At most you're telling me your AMI costs half of NGINX's... Which still is a lot more than just running Ubuntu with a free AMI.
This is true, but the same principle applies to EC2 instance. We also tried renting an m3.medium and installed Debian+NGINX on one and Unikraft+NGINX on another and experienced the same cost savings with, in addition, twice the performance (roughly 12K req/s vs. 26K req/s).
i've always thought this is collecting pennies in front of a steam roller: you save 50% only to go out of business when a black swan hits and you can't debug an issue in production in any way. i guess this changed?
There are many ways to debug unikernels in production, particularly, we have in-kernel monitoring (e.g. ability to enable fine-grain memory, scheduling, and resource monitors. These work also any included libraries). You can also produce flame-graphs of the entire unikernel image which show the entire kernel stack and the application. There is also remote GDB support (which come with additional scripting tools for automation) for debug-images which are built along-side the production image.
There is one primary reason why large companies are migrating to AWS Azure etc that most people seem to be missing here.
It’s outsourcing the blame. This is the key to boardroom success. When something goes wrong in their own data centers, the CXOs need to shoulder the blame.
Large corporations outsource everything, including the guy who cleans the copiers, copier not working, blame the maintenance vendor.
Something went awry with the infrastructure, blame AWS
App development project went down the tubes? blame the contractors & outsourcing companies.
that is an argument I have seen before and it is generally true: the core idea is for the corporate entity to operate a riskless arbitrage where all risks are shouldered elsewhere, ideally where they are less understood and thus underpriced.
the problem with applying this concept here is the hyperscalers are not exactly a counterparty likely to underprice IT risks. in any case is there any public evidence of this "blame transfer" in action (pressumably with associated remedies)
Surprised Cloudflare was mentioned so late in the article and almost a footnote. I think that's not representative of its impact of their strategy on the industry.
Judging just from the headline, I assumed the article would mainly be about Cloudflare's entry into that market. The whole point of their strategy has been to position themselves as the foremost middle-men of the Web and then start eating all the service providers "behind" them, including and perhaps especially cloud hosts. They've set everything up and now they're executing on that plan, and things are starting to get interesting.
I currently work for a large fortune 50 company as a Senior SRE in charge of server build automation for both cloud and on-prem. We have found that on-prem is cheaper and has less hoops to jump through. We do use vRealize Automation and Orchestration from VMware and it is a good agnostic platform to do this.
Even when adding the upfront investment costs of hardware and ongoing cost of qualified personnel and support duties? And what if overnight your company needs 10x the capacity?
We actually are setup with a major hardware vendor where they roll out that 10x capacity ahead of time, and we only pay for it when we flick on the switch. Otherwise, it is powered off/Standby just sitting there drawing minimal power resources if any.
This is good competition is needed to help with pricing. AWS and other large cloud providers have an incentive to “lock” customers in and raise prices.
There must be a pressure by competition to keep this incentive in check.
I would say this market is due for some consolidation. AWS has basically has enjoyed a nice market share for the last fifteen or so years. In recent years MS and Google have become credible alternatives and in addition to that there are now numerous smaller players that you could feasibly use as well.
The key challenges in this space are standardization of commodity features, making the whole business of using any of these solutions less complex, and driving cost down. This goes against how the big three are making of course which is exactly through vendor locking aggravated by complexity. Once you figure out how to use this stuff, you have a lot of non portable stuff depending on that vendor. Replacing that costs money and is never a priority. That's why AWS stuff can be such a PITA to deal with. It actually ensures customers can't move away that easily.
However, the common feature set shared by most of these cloud hosting solutions (running vms, databases, docker, load balancers, networking, etc.) is a combination of expensive to use and still too hard to manage. So, the game for new players in the market is being more convenient and less expensive to use. Ultimately private cloud and the cost of getting started as a PAAS provider getting lower means that this is a race to the bottom. Most of the smaller providers already lean heavily on OSS solutions.
The question is whether the big three can long term afford to not support the emerging standard stuff. For example at least for me all the convoluted proprietary AWS stuff is actually increasingly a reason for me to avoid depending on them. It was great when they were the only company with decent solutions but that is increasingly less true.
The choice of running in prem, or even changing cloud vendor is diminishing.
Yes if you only use compute and storage moving is easy.
If you want scaling in AWS then move to Azure I think that would involve a lot of changes (as it would the other way) I am not sure how much of it Terraform can translate.
I would guess K8 can do this is a vendor neutral way but I am not sure since I have never worked with it.
I do know for a fact that getting your devops pipeline set up in AWS
is really complicated. We ended up having to hire two "elite" AWS
consultants for 14 days to get it all working.
I was not involved directly in it, but my team had great need of the finished pipeline. If we went with Azure we would have to do it all over again.
But most of our stuff is in .Net so getting it all working o Azure would have been magnitudes better.
The more "premium"/ high level services one starts using the ability to move on prem
or switch cloud becomes harder and harder. Which is of course the goal for the cloud vendors.
Speaking from experience, just because Azure is Microsoft doesn't mean that it's point-and-click to get a .NET app running on Azure, especially if you do anything non-standard in startup. Just google "ANCM in-process failure" to see dozens - hundreds - of out-of-the-box .NET apps that just refuse to run in an out-of-box Azure compute/VM.
To develop a canonical model over the infrastructure that can be reused between cloud providers takes a lot of knowledge. My experience from Terraform and Azure is that it is not in general worth it.
I ,for one, welcome our new IT overlords, the AWS/GCP/Azure. The old overlords of IT, the IT infrastructure depts, were too slow and bureacratic and lacked elasticity. For example, if I wanted to spin up a few servers with Cassandra -- I would have to undergo rounds of review for approval of a new technology then work on requisitioning the h/w or VM, and figure out support chain-of-escalation. Also somehow , on-premise VM were always crashing or running out or disk space.
With the new overlords, I have to pass through fewer reviews (I work in financial sector; we love review meetings and gates.) and provisioning a service is as simple as a AWS/GCP/Azure url. And tearing down stuff is easier. The old overlords of IT would be pissed if we decided we did not need a h/w server and we would have to fill out questionaires to determine how we could have been so misguided and then I had to wear my dunce hat for any meeting we would have for the next requisition.
I love the battle of cloud computing. With the big three, they are still hold too much power. I look forward to the day when Vultr, DigitalOcean, Ovh, Linode can get approved to be used in my employer's IT stack. With Terraform or Kubernetes, I would be able change our dev/qa/uat cloud environments at will and this will force the Big Three to lower their prices even more.
It's interesting that they decided setting up a cloud server did not require the same level of review. That does not seem exactly like the result of a technical difference.
When you bring in a new technology or capacity to on-premise, you would want to maximize the bang for the buck which means that you want to review your roadmaps to see who else could benefit from the new technology or capaciy. For Cloud services, this is not really required. Also, there is a level of trust with the Big Three meaning that we can trust that they will be able to manage SQL,Cassandra etc with ease; this IT departmens do not have the manpower bandwidth to keep on adding new servers or services. Hence some of reveiws are not really required.
The difference is that your local IT team is overworked, understaffed (yay IT is a cost center), and bears the full deployment and ongoing monitoring and maintenance responsibility.
In new cloud land you're essentially contracting out a team at Amazon to handle all that. You can rent a team that is dedicated to that one particular service. In cloud land behind every service there is a whole team of people dedicated to maintaining that particular service, in on-prem land behind every service is a 1/20th of the capacity of your IT team.
OS currency is a big deal for us in Banking. The Cloud vendor maintaining the OS currency and patches will save us so much headaches. Its entirely possible that version x of a custom application requires version q of the OS but the Cloud vendor is mandating version r of the OS. In this case we get to go back to business and app teams and force them to either upgrade their software or containerized or retire their app. Without this stick, we would be stuck with apps untouched , not- maintained and vulnerable forever.
I doubt it will save anything, I would say the lack of flexibility will highly likely cause massive headaches as applications just suddenly stop working, as will having patches forced on you.
This stuff has nothing to do with cloud but mainly on how the business thinks off and controls IT.
Some finance businesses have the resourses to hand to compete with the cloud providers or at least create internal self managed equivalent
It's also about unreasonable productivity of APIs. EC2 (with autoscaling groups) + VPC + S3 + SQS + DynamoDB can blow the productivity of most IT department out of water. Let's face it: most of engineers have no clue or stamina to build something even remotely resembling S3 or SQS.
250 comments
[ 2.9 ms ] story [ 260 ms ] threadReady for testing and pre-release R&D.
I was on the development team for a couple years; now, I'm independently developing some technology on the platform. I'll announce it soon, I hope.
With the recent AWS outages, and systematic de-platforming of some legal businesses, there seems to be a growing realization that dependence on centralized hosting platforms is ... dangerous.
Holo / Holochain seems to provide a unique solution to some of these problems.
Why not "own" AWS, GCP, Azure etc.? Because you can't. They are part of conglomerates that come with a moral baggage that is too heavy for some.
ps. No "hyperscaling" ability required. The idea that business activity is either complete dominance or death is part of that unfit-for-purpose mentality.
That seems like a common subset that would satisfy a lot of use cases with minimal ceremony.
I think, a VM approach would have been a bit more flexible.
How do you even begin to operate a consistent security model across "8.4 million global data centers"?
> Auditors on the Akash Network review cloud providers and digitally sign the provider on-chain with their certificate. If you only accept bids from audited providers this means you are trusting the Auditor/Provider not just a Provider. [1]
For a business at scale, I don't see this being solid contractual ground. Now I have to trust Auditors to continuously vet Providers? Simply too tenuous for my comfort.
[0] https://akash.network/about#vision
[1] https://docs.akash.network/glossary/security
https://www.zdnet.com/article/why-does-the-it-industry-conti...
https://www.horsesforsources.com/gartner_fail_automation-AI_...
The Gartner reports were complete and utter garbage, filled with factual errors. Looks like they were written by interns, regurgitating marketing material - completely removed from reality.
Since then, whenever I see somebody mention a Gartner report, I immediately dismiss that part of the argument.
> The second report ("1999 OS forecast: The Linux face-off") projects market share in server dollars for the next few years. This one even contains a pretty graphic. The Gartner Group finding is that Linux will fade from the scene following the release of the first service pack for Windows 2000.
> The third report ("Red Hat's future: Boxed in") concludes that Red Hat will fail to overthrow Microsoft by 2004, and suggests that Red Hat needs to find a more viable business model.
If you get a report from them on how the pitfals of setting Oracle's Spatial, it's probably good. If you get a cost benefit comparison of Oracle and Postgres, it's certainly trash. (Who am I kidding? They still insist Postgres doesn't exist.)
That said, none of the documents are worth the price you pay and the hassle of interacting with them.
Basically analysts mash up what vendors and customers tell them what they think. They have conferences with large enterprise customers and vendors sponsor those conferences. So there’s a element of indirect pay to play. Many companies use them to review contracts as well, so they know what kinds of deals are out there.
If you expect to understand technology from Gartner reports, that is dumb. But if you want to understand the market and offerings for large commercial/gov enterprise customers, they are a great resource. If you are a startup, you should not pay any attention to Gartner unless you are selling in that space.
In the case of this topic, the information that’s key to understand is that Gartner is getting feedback that their customers are adopting cloud services, but are seeking to optimize the spending, including with some of the smaller players in niche areas.
They are pretty shitty for that too. They only get anything passable if you are after the offerings of the specific companies that pay them for the publicity.
Yeah I distantly remember "on prem", too. Does everyone, everywhere really need to be equipped to hyperscale at a moment's notice? Wouldn't it be better not to be beholden to a megacorp that serves you with 30 pages of opaque billing for business-critcal functionality? Doesn't hiring external consultants to manage your fucking cloud infrastructure seem like sort of a risk as compared to keeping the services-wrangler in-house?
...I don't have an MBA, so these are half-serious questions. If anyone can chime in with (at least) a half-serious answer, I'd halfway appreciate it. I understand that The Economist is the voice of global capital, and global capital has decided that there's money in them thar clouds, but still!
Traditional corporate datacenters were dogshit when it came to provisioning new resources for business capabilities.
In today's world, we need the ability to spin up and down deployments very quickly. We don't need huge numbers of resources, just fast response.
People are investing in cloud because its faster, more agile and better, not because it's cheaper.
The top line looks expensive, but if you then do the correct analysis on the cost side of the equation (taking out staff costs and DC costs) then I think it is either close or in the black from a cost perspective.
If the business case was really bad, I don't think we would see the rapid uptake we have seen.
Then you have competition (GCP, Azure, Alibaba) which will tend to lower the price, but for now the cloud industry grows. Their goal is to attract new customers who didn't use the cloud before.
And the ones we got on Friday were wrong (we asked for 3 * 16GB ones + 1 * 2GB, we got a 16GB project container with 4GB pod limits).
For our hadoop request - yes it got provisioned in >8hrs (mostly I think because no one is using it) but none of the access instructions work and everyone who has used it is mysteriously unavailable - so we can't access it.
Compare and contrast to self service from any cloud.
Why : well it's demand management. If you do self service and say "pay for what you want" then IT's costs bloom and the CIO is shot. If you do that in a cloud scenario then Cloud costs bloom and the CIO says "you guys have spent your money badly". This means that you need police and you need people who you ask to give you what you need if they are happy. These people obstruct you and then get things wrong.
First, there are internal processes. All too often they end up needlessly slowing velocity. This isn't a technology problem but a people problem. Using a public cloud enables people to circumvent these issues, in some cases.
Second, there are costs. What does it cost to give someone the features they ask for in the time they want it? This is a requirements, design, time, and cost problem.
Basically small markets (internal IT) can't sustain the investment to support the requirements design time cost issue - while the public cloud folk can say "any colour you like so long as it's black" and there are enough takers to pay for it all.
Do you really? In my experience, admittedly outside of SV, pretty much nobody needs that.
That only makes sense to me in the context of a business being an IT shop to begin with. Most businesses aren't, and their new product offerings require factories and dies and labor and materials. Besides which, the problem of provisioning is entirely overblown for this scale of business. I can spin up new servers all day on our on-prem vSphere cluster.
Lots of businesses outside of 'IT shops' use computers. I currently work in Civil Engineering and more and more of our product and service offerings have an IT/Online/Web based component which involves setting up some sort server.
I can spin up new servers all day on our on-prem vSphere cluster.
I what price? At my old job an on-perm vSphere 'machine' cost my department a lot more than what AWS was charging.
At that scale in a single company the ability to provision verticals weighs heavily in architecture and design choices.
"Should we spin this new feature out as a microservice or fold it into an existing app so we don't have to wait weeks and do crazy cost accounting for new environments?" And the answer has follow-on effects because now it has to be accounted for in a specific teams' backlog and we encounter yet more waiting.
And since it's my 'private' cluster I don't have to worry about queuing my jobs and waiting for the machines to be free. There is no way my department would have bought me those machines as physical hardware.
If you don't meet those criteria, unless you are absolutely horrible at negotiating with your vendors, you aren't saving money going into the cloud. I don't care that "datacenters are expensive to build". If you are large enough to need to build your own datacenter instead of renting floorspace for pennies on the dollar from Switch or Equinix (although Equinix is kind of pricey) or Windstream or insert provider - then the datacenter is a rounding error in your 10-year IT budget.
The problem is "I moved company XYZ to the cloud" is now a requirement for an IT exec to show up to the country club on Saturday. So whether it makes any fiscal sense at all, the board wants to know "what our cloud plans are" and most execs are more than happy to oblige in order to pad that resume. In my experience, most of them leave long before the bill comes due, and people start asking why the IT budget quadrupled in 3 years. The best part is, most of these companies can't pass along that increased cost to end-customers like a Netflix can because they aren't actually making money on the apps they're hosting in the cloud, those are all internal business facing apps.
Money is ALWAYS the issue. Whether you can pass along increased cost to your customers or not is entirely dependent on what your business is and the competitive landscape of it. I don't care how "agile" the cloud makes you, if your cost of IT is more expensive than the revenue you bring in, you're bankrupt. It's the running joke around here about people who build autoscaling k8s based infrastructure for a site that could be hosted on a $5/month VPS with even a little bit of tuning and caching. Complexity for the sake of complexity is called stupidity unless you're doing it as part of an educational experience. Going to production with needless complexity is a great way to ruin a good idea.
>Cost is not the same as value.
I guess it's a good thing I didn't state that cost is the same thing as value?
I think the goal is that it should be the same amount of engineering time to set up both options, and both should cost $5 per month.
Today, that is broadly untrue. Although a few niches like appengine offer this.
With operating your own DC, you have to hire really great people and hope they don’t fuck up. You have to plan much in advance for capacity changes. It’s very much possible that something goes wrong in your compute infrastructure and simply nobody in your team has the expertise to fix it. Put it another way: can you pay/hire the same kinds of engineers that work at AWS/GCP? If not, cloud is the safer route. Cost does matter, and this is why companies sign multi year contracts to get discounts. But business continuity is critical.
But if you need a server then (in most companies big and small) it's far easier to spin up a VPS in the cloud than deal with internal IT.
> "I didn't state that cost is the same thing as value?"
You're arguing over cost, but the cloud is often much greater value compared to traditional IT. Spending 2x for 5x more value is a valid decision. If it doesn't work in your situation then don't do it, but it's not the same answer for everyone.
Money is COMPLICATED and THE issue.
Money is complicated because you have income and expenses. Time to market and velocity can help you get more income. This needs to be balanced by expenses.
When you're growing or your needs are uncertain than a hyperscaler can help you get moving quickly. This aids in time to market. But, you do this at a cost. Hyperscalers cost more than running it all yourself.
If your workloads are steady state or changing at a predictable rate than it can be cost effective to run your own infrastructure.
There are other elements to all of this such as global location of hardware compared compared to end users.
Money is the issue because companies are in business to make money. Sometimes companies take a loss now in order to make money later but it's always about the money.
You're painting the picture as if AWS is the 2nd version of the IBM saying: "No one ever got fired for using AWS". This is simply not true...at least not the part where "IBM" (i.e. AWS now) is somehow inferior to the competition.
> If you are large enough to need to build your own datacenter instead of renting floorspace for pennies on the dollar from Switch or Equinix (although Equinix is kind of pricey) or Windstream or insert provider - then the datacenter is a rounding error in your 10-year IT budget.
Then you clearly don't understand the TCO of using a data center vs cloud. The datacenter cost itself isn't the cost - the people who operate it are.
On a computing resource basis can you operate SQL Server on a rack in Equinix for cheaper than that in AWS? Sure, but who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.
If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.
P.S. - For the record, many large internal IT groups do in fact do simple lift and shifts from traditional data centers and don't fully realize the value of cloud computing resources. However, these companies will usually mature eventually and evolve services to ones that are more advantageous (i.e. setup SQL Server on EC2 and the migrate to RDS). AWS (et al) have the ability to do this with ease.
Who cares? This is IT and is not relevant. And they have Brent and he will solve everything.
Sure, but you can go from "I have 5 people managing my 150 instances of SQL Server" to "I have 1 AWS sys admin". The point wasn't that the costs are eliminated, it's that they are baked in.
I think Cloud still costs more overall, never been convinced it actually saves money. I do think it tends to increase developer velocity though, and that can be worth it.
This is especially true for more than basic services. If all you need is a bunch of Linux servers, sure, you can find people who will keep them patched for you without too much trouble (although it'll almost certainly turn that “rounding error” into a serious number) but once you start building things which are higher level with legitimately hard problems to solve and need to have good solutions for all of the monitoring, security, ops, etc. needs that staffing becomes harder and you're going to have to sell your executives on the idea that the best use of your expensive A-team is cloning S3, Lambda, etc. when you can get that benefit immediately without a massive up-front payment and have the same people working on a strategic benefit.
No need to clone; all of this already exists. Take a look at CNCF.
* Kubernetes
* High-level storage services: Ceph, Cassandra, NFS
* Lower-level storage for whatever you pick for the previous point
Each of those, and the combined service you build, needs staffing for operations, monitoring, security, etc. support. If you follow any security standards or are in a regulated industry, you're on the hook for documenting compliance on all of those, too, especially if you need to be able to demonstrate things like immutability.
You certainly can do this but that's a number of full-time jobs requiring expertise. If you have enough storage usage you can potentially justify the savings over the rates you'd be able to negotiate but that's a big up-front cost which you hope but are not guaranteed to recoup.
I took your previous comment to be about on-prem cloud technology. It exists and does not need to be "cloned". If you were taking about staffing resources, we've pretty well established you're going to need staffing to manage both cloud and on-prem tech.
You need to document whether it's on-prem or in the cloud.
If you buy storage by the petabyte or use a large enough amount of network egress, you can definitely make that money back but it's a major commitment.
Agree, I don't think anyone is buying SANs that don't need them. If you're spending 7 figures on your AWS bill, you should be looking into other options.
That is building an S3 replacement. You aren't writing every line of code personally but you are selecting components, configuring them, operating the service, and testing it, which is a major commitment. Simply running k8s at an equivalent level is a non-trivial commitment.
> Agree, I don't think anyone is buying SANs that don't need them. If you're spending 7 figures on your AWS bill, you should be looking into other options.
Note that the 7 figure commitment I mentioned was what you'd need to build an S3 equivalent. Simply having 24x7 staffing and hardware easily puts you in that range.
You're casting this as a much more complicated concept than it is. It is essentially the same old "build vs buy" argument. No, it does not make sense to manage a datacenter to run a single wordpress blog, nor does it make sense for a telco to run their infra on top of a public cloud. These are all truisms.
My point stands that if you have a seven-figure AWS bill, you already have plenty of associated staffing costs. Once you hit the break-even point between the two, you can decide whether you want to continue to pay linearly, or go down the path of increasing RoI.
I've seen this "even a lift-and-shift reduces TCO" argument before, but it only ever works if you pretend that AWS lives up to its zero-management hype. Maybe, for someone, it does. Not for any deployment I've seen. This never prevents management from pretending, of course.
EDIT: ah, I've seen that the argument has retreated to "it reduces IT headcount." Again, maybe for someone, but I have yet to see it actually work, and every time management has just hustled the TCO calculation.
Who is saying zero-management? If you have a consultant lying to you, that's not a cloud problem and I've heard the same people lie about all kinds of on-premise services as well.
What I've heard the major cloud providers say is true: you need _less_ staff and you can spend time on different things, which has been generally true. For example, instead of needing to configure a bunch of network hardware (which is a distinct skill you need to hire) you can have a general devops person use Terraform to manage routes, firewall rules, etc. without also needing to be a Cisco expert and spending time operating the underlying hardware.
The key question to ask is where your infrastructure was to start with. If you have an A-grade ops team, a lift-and-shift will probably not be much of a cost savings because you presumably already have economies of scale working in your favor, unless you have enough bursty workloads to pay for it that way. The catch is that most places do not have that high-end ops capacity so what you're really doing is taking your limited IT capacity and reclaiming the time which is currently going into things like managing Dell firmware updates.
You're implicitly arguing that NoOps is not only a functional model, but also scalable and much less costly than having a {Dev|Sec}Ops staff. The last time this was publicly debated between Adrian Cockcroft and John Allspaw, it was immediately obvious to everyone that there is no such thing as NoOps. You can have developers support their own apps in production on the cloud, but that takes away time from developers and comes with its own set of downsides (most prominently, a disinterest in what they consider mundane or boring). Once your developers are spending, in aggregate, more than 40 hours a week do ops work, you'll gain efficiency by hiring a DevOps Engineer whose entire career focus is all the things developers aren't typically great at. I've seen this in-person as an employee at medium, large, and a Fortune 50 business. I haven't yet seen a single app, not even serverless, that requires zero maintenance. I have also seen highly-distributed, performant apps run on cheap hardware with no SAN or VMware to drive up the cost and complexity. That same app, made up of mostly microservices, was much more expensive to run on on the cloud.
You might want to stop throwing around terms like TCO when you also clearly don't understand how to calculate it.
I'm starting to be of the mindset that us-east-1 is already "Too Big To Fail". I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.
I'm not. You implied too much.
> You might want to stop throwing around terms like TCO when you also clearly don't understand how to calculate it.
I do. I never said that people were completely eliminated (see one of my child comments). Everything you implied was that "well devs are doing the effort and that doesn't get calculated". That wasn't my argument - you just added that nuance as a retort.
> I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.
I guarantee you they are not. Also, how many data center providers let you simply spin up your whole environment into another region, point n click, and complete it in hours?
Source - work with 100+ companies with PE folks that sit on boards.
I'm speaking from firsthand knowledge. How embarrassing for you to think that because you know some Private Equity folks (generally the worst people at managing technology), that 100% of the rest of the world is doing (or not doing) exactly what they're doing.
You wrote paragraphs about how expensive it is to hire people to do stuff in a datacenter. What is your argument, if it is not that this is more costly than outsourcing it?
This entire debate should be recast the simple "build vs buy" concept that it already is.
So am I. My reference to PE firms was that my anecdote of the few boards I'm on is supplemented by their experience.
> This entire debate should be recast the simple "build vs buy" concept that it already is.
Because that's entirely the premise of my argument. Did you read the parent comment and what I was responding to?
PS - Say what you want about PE folk, but they are the most ruthless capitalists. So IMHO they are far more incentivized to drive cost efficiencies than anyone else because profit = EBITDA = valuations.
If you're not blasting your AWS account manager and asking what their plans are to avoid this happening in the future, then your CTO is not doing their job.
If AWS itself is not internally working out how to "get off us-east-1" for its global services, then it is failing in its role as infra provider.
AWS has a lot of stuff about how to make your infra redundant and resilient to failure, in their environment, mostly about replicating across AZs and then across regions.
The fact that their "global" services are not actually "global" but are "everything goes to us-east-1" is broken and if they're not fixing it, then that is when you start looking at other alternatives.
I guess I find that a bit humorous. I've been architecting solutions that span on-premises and cloud for nearly a decade so I'm intimately familiar with both. I don't actually have a horse in the race. You want cloud, buy cloud, you want on-prem, buy on-prem. But the marketing about one-size-fits-all is EXTREMELY tiresome. And the idea that every single business can or should be moving their entire infrastructure to the cloud is as well.
The fact you imply that somehow you don't need people to manage the cloud makes me seriously question what at-scale deployments you've been a part of. I have yet to see a single enterprise customer save money on staff by moving to the cloud. More often than not what they end up doing is laying off the "tape jockey" and replacing him with an SRE who costs 3x the salary. Congratulations? Now you're paying more for an SRE, and you're backing up to S3 at about 5x the cost of what you were spending on tape? But I guess the 7 years of backups you're legally obligated to keep are more agile?
>If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.
I don't think the concept of cloud computing is lift and shift and never said it was. But that's not really relevant to OP's question, what is relevant is what's happening on the ground.
What's actually happening is that most established businesses "moving to the cloud" are lifting and shifting and finding out the hard way. I'm not saying that's what they should do, I'm not saying that's at all a good idea, I'm telling you that is what is happening. And if you think "well they'll mature into a more cloud friendly model" - you're as delusional as their management. What ACTUALLY happens is after they lift and shift and get a year's worth of those bills, they lift and shift the stuff that never should've been in the cloud in the first place back on-premises. Hopefully by that point they've learned their lesson and have a thoughtful evaluation of their applications: re-home, re-write, leave in place.
100% agree. Didn't mean to imply this. As with anything....it depends.
Most data center machines run at a tiny fraction of their capabilities; seeing a CPU running at 25% is definitely something that stands out. Because of this, cloud operators can oversubscribe these machines in various ways, packing workloads of multiple tenants onto a single box while a single-tenant bespoke DC can't. They can also do things that schedule DC-wide resources (namely power) in ways that renting space in a colo cannot; they can shed load in insightful ways when a power overtopping event happens that can minimize the impact (e.g. throttle / nuke VMs that can operate in a degraded mode before impacting other critical services). No hetergeneous, multiple-private tenant DC is going to oversubscribe their rack- or row-level power delivery to tenants and require them to hook in to a smart system to manage overtopping events; they just won't trust their tenants to do the right thing, as they have no power over the tenants' machines and power draws; the cloud providers do.
At the end of the day, the economies of scale are just hard to beat, in terms of COGS.
The real question is whether they'll pass the savings from this on to the customers or the shareholders' dividends, and so far it seems like it is the latter.
Are you familiar with VMware?
Kubernetes fixes that, as does VMware. They can both distribute workloads efficiently to ensure good utilization. VMware has done that for 20 years.
Another thing is costs of building something new. In the old world it would mean doing the math on resource needs now and in the future, since adding Hw meant capital expenditure. No such problem now and its easier to get started, see if it works, and scale or kill early.
And of course as others have mentioned, keeping it all running requires expensive expertise. Esp. now with the internet that looks like automated wild west. You cant release a service without it being auto-probed and auto-attacked within 2-3hrs.
Nope. With the cloud, the costs have went down for any large coMpany, even those that just moved the old crap to VMs running on someone elses hw.
Are you implying that there isn't idle cloud hardware?
I think I found your problem...
Compute on preemptable VMs/spot instances is cheaper in cloud.
Object storage, if you would have required a vendor support contract on prem, up to a PB or so.
managed kubernetes, if you would have required a vendor support contract on prem.
I entirely associate fantasy football with working hours. Ditto a variety of other kinds of sports betting. Like, I know people do these at home, too, but there's a lot of it happening during business hours, at offices.
I know there are also a bunch of requirements for where you can store user data these days due to privacy regulations.
Agility, flexibility, security, global deployments, and consolidated billing and operations are far more important.
But the others are spot on.
Going into the cloud does not alleviate any security decision. You can't buy security.
Of course you can. In this case, many of those decisions are already made for you in the cloud.
A simple example would be using RDS for a database service. It's secure by default and only the credentials are an issue, compared to running a database of your own somewhere internally that may be exposed in dozens of ways. Less decisions, less surface area, more documented reference setups to follow, more scanning and analysis tools means better overall security posture with less work.
It would have been much, much cheaper to rent gpu in the cloud and when we knew _something_ about our workload we could buy an on prem and maybe rent extra if needed.
But we weren’t cleared for cloud.
Regarding in-house expertise: I think most companies would like to have in house expertise but they can’t hire the expertise they need so they use consultants.
However... I was also sort of surprised, because last time I priced this out, you can have pretty low utilization rates of the equipment and still come out ahead.
Don't expect these ratios to be authoritative; there are obviously different economies of scale and many many details I papered over, but this should just give you an idea of the back of the envelope "relative costs"; just that 5x is way out of the range you should have guessed.
Mostly because they simply didn't know how to run an IT office, and so they couldn't build a culture where good IT people want to work. When they finally get a strong candidate, they end up overloading the poor hire, because finally, it looks like they can get their infrastructure fixed or dream project off the ground.
I used to work at one of the most popular fast food chains. I had one project that was internal to them and one that was customer facing. The internal product, a search engine for documentation ran on-prem while the e-commerce store ran on AWS. Running the former on-prem was just fine.
I'm sure glad we were on AWS for the customer-facing site, though. We'd run ads during the Macy's day parade, college national championship, etc... dear lord I could not imagine wrangling the necessary people to scale that out from an infrastructure perspective. It was very nice just doubling the number on our ASG, failing over to the secondary database, launching another DB instance with double the capacity, and repeating for the second one.
Between those popular ads and DDoS attacks I'd say AWS was a win there, but internal tools don't benefit from the cloud as much. Also not an MBA, but I feel like that's a rule a thumb you can add to the repertoire.
https://www.businessinsider.com/bank-of-americas-350-million...
It seems no company wants to be in the business of in house IT Systems. They think moving all their infra into the Cloud absolves them of all the responsibility of owning it themselves, they don't give a shit how much it costs but then tell you everyone has got to save money and tighten their belts.
I've done this: built a 17 server cluster, housed in a colo, for a high compute startup producing 3D avatars from photos automatically. The hardware for the cluster, including a Federal Reserve class hardware firewall, was just over $50K, with a month to month colo bill of $600. This was in contrast to $96K per month if the same configuration were at AWS. This was 7 years ago, so adjust the prices, but you get the exponential higher expense point.
A private network is ridiculously easy to set up. In our case, we had two switches, one public, one private. Every machine had 4 NICs. Just bind some to public addresses, some to private, plug in to appropriate switch.
Unless your traffic is extremely bursty, auto scaling is probably more trouble than it’s worth. We were serving millions of users with three dual-socket machines. And if you really need it, you can try to handle base load with dedicated servers, and bursts with AWS infra. But at <$10k/mo on AWS, you’re not getting anywhere near the horsepower I’m talking about.
I understand it was a startup and may not have needed high-uptime ops from day 1, but a lot of organizations do, and the cost is not that dissimilar.
You also have to include the human time spent setting up and maintaining those servers. And also the fact that you can probably use a managed service on request instead of 17 servers running constantly. Not even mentioning redundancy.
Most businesses are not technology businesses, they deliver goods or services to their customers where IT is part of their costs to deliver.
Investing in IT people to run that bare metal is a huge expense that doesn't improve your goods or services directly.
As for "where a task would floor 32 cores for 3.5 seconds"... means that your internal IT infra had to support that maximum requirement headroom 100% of the time, even though it may not be used for 99% of the time (depending on load patterns etc).
Enterprises would rather not maintain inhouse infrastructure for many of the same reasons they engage MSPs despite persistently shoddy services.
Building up an IT infrastructure that can provide services at roughly the same quality and cost as AWS takes a large investment in talent and infrastructure. And many companies don't see the value in making that huge upfront investment for something they might not need. Better to start with AWS, and once you get to the point where you can lower your costs by 50-70%, then start looking at moving in house.
no, you missed an important point here.. no one, at any scale, has been able to compete directly with AWS on the combination of service, scale and cost per unit.. its intensely competitive
I will say that if your work load is relatively fixed in magnitude and especially if it's bandwidth intensive, you can probably beat the "big cloud" providers by at least an order of magnitude. If it's extremely bandwidth intensive you should never consider mainstream cloud as outbound bandwidth is absurdly overpriced... like 1000% markup or even more. I can't exaggerate how insane bandwidth pricing is.
If you are just prototyping something, your work load is highly elastic, or your product is rapidly evolving, the flexibility and outsourced sysadmin of cloud will probably make sense.
Edit:
There are edge cases too. If your work load is, say, inbound bandwidth and storage intensive, AWS might be great since S3 is cheap and inbound bandwidth is free. Just keep in mind that you might pay a lot for egress if you ever need that data out. Will you?
Run your own numbers and be detailed!
It was quaint, adorable, personal, and hilariously inefficient.
It seems very clear to me that these kinds of businesses are much better served by cloud. No one of them is that valuable of a customer, but there are so many.
Companies that are large enough to have an IT department with multiple highly skilled professionals onsite, or who could even contemplate doing a real data center (most "on prem" is in a closet), I agree is a different calculation.
To operate a production cloud you need expertise at all parts of the stack. This includes network, kernel, maybe firmware, orchestration, etc. You also need adequate planning where your on-prem data center teams continuously upgrade your hardware/firmware and add capacity. From an organizational perspective this is an entire VP or SVP level organization depending on where you are.
Operating large infrastructure is hard. If it’s not your core competency there is really no point in dealing with it. Either way, it takes enormous effort to hire and keep a team skilled to operate infrastructure. You will lose your team to big-tech eventually.
You don’t need external consultants to manage your cloud infrastructure. Your existing engineers can skill up to use it. Public cloud will commoditize what typically takes a lot of time to plan for capacity, DNS, networking, scaling, etc.
In short - its prioritizing speed of execution over everything else. The opaque billing is just cost of doing business. But unless tech is your core competency, opaque billing is much easier to deal with than all the other situations that come up with managing and maintaining your own infrastructure.
The question for businesses comes down to economics, of capital expenditure vs operating expenditure. If I buy my own hardware, I have to spend capital and depreciate it as part of the operating expenses. If I lease hardware (whether on prem or cloud), then that's pure operating expense.
There are accounting and tax implications. There's also the "stick to your knitting" of a business. Do I need my own payroll service, or do I outsource it. Do I need my own accounting, or do I outsource it. Do I need my own cleaning service...
Solution: Subscribe to a middle-man that manages it for you.
Problem: The middle-man is slowly turning the screws to increase its profits.
Solution: Subscribe to another middle-man to figure out how to decrease the first middle-man's profits.
It seems to be a universal truth that computing people try to solve all problems with another level of indirection.
For most, this was never the effective reason.
Most rationales behind cloud migrations are likely based on quick scaling, which is meaningless in most cases - except for the signalling of committing to a future where demand hockey-sticks upwards.
It's valid if you are netflix, but how many netflixes exist in the world?
This isn’t about computing but about specialization. Should McDonalds be in the business of operating Data Centers? Or should they just buy it from those who do it best and focus on selling food?
Should they be in the business of cleaning bathrooms, or should they just outsource that to a temp agency? Sometimes specialization pays, and sometimes it doesn't. Perhaps it doesn't make sense for McDonalds to run its own web and corporate servers, but doing so might make a lot of sense for more tech-centered companies. And I'd guess a server farm is miniscule compared to the size and complexity of their ingredient supply chains.
Uh, they charge more per byte, that's not what's going on. I don't know if that was AWS's Garman being deliberately misleading, or the Economist journalist misunderstanding.
That's not accurate because many businesses do migrate for cost reasons. Especially for internal non-public facing servers like email. E.g. They moved from self-hosted Microsoft Exchange servers to cloud-managed email like Office 365 Exchange Online -- or switch to another cloud competitor like G Suite.
>Ms Wang and Mr Casado have suggested that firms should think about building their own private clouds to keep costs down. So far few firms have opted for such “repatriation”, which is both costly and makes it harder for businesses to enjoy the benefits of essentially unlimited computing resources in the public cloud.
As a case study, The Guardian newspaper tried building their own private cloud and failed. They ultimately just went with AWS: https://web.archive.org/web/20160319022029/https://www.compu...
If the company's business profit is not in the tech sector (like The Guardian), this means its IT datacenter operations is not its core competency which makes it hard to build internal IT that can keep up with AWS capabilities. Tech businesses like Dropbox, Facebook, can hire & maintain skilled IT staff like AWS but most other non-tech businesses can't.
What I see often is that it's hard and costly to replicate the whole cloud experience in-house. OpenStack is a behemoth to operate. However, if you build something more specific to your needs (and often mix in selected public cloud services), you'll get a much cheaper stack.
Small businesses can often run very efficiently with a few rented physical servers in a colo, G Suite and AWS S3.
Larger ones can provide a platform specific to the technologies used in-house and combine with public cloud for any other tech stacks.
Operating a small ceph cluster is almost a full time job, at least if you care about your data...
I guess the equation changes when you have more data or looser requirements. That's actually my point. It's very hard to recreate the whole AWS experience, but it's quite possible to pick any piece and do it much cheaper.
Think Urbit but for sysadmins and you pretty much get the vibe.
At the time I last tried it (some two or three years ago):
- most of the documentation was written assuming an rpm-based distro and not Ubuntu or Debian (I remember especially having had trouble with subtle differences in iptables/nftables between the distributions)
- on top of that it was right in the middle of a python2 => 3 migration with the result of shit breaking left and right
- it was/still is written in Python which means a second or more for every CLI command to initialize, much less actually do something
- half the instructions in the documentation seem to never have been actually tested, which combined with the two points above made for a really punishing "experience", especially with more rarely-used features such as k8s-in-openstack
- the documentation itself still is (just looked it up) badly organized, with you needing to read four guides (Install, User, Configuration, Ops/Administration) per OpenStack component. Ideally, there should only be one guide for Installation/Upgrade (including Configuration and everything from Ops/Admin) that has everything in it to get all supported parts of the component up and running, one User's Guide that shows how the component is used and what the best practices are, and one Troubleshooting guide.
If you can set it up into a working configuration, it's a nice project for sure. But the road to that point is rough, and you will likely not even want to think about an upgrade simply because of how complex it is.
It worked like this - you had to fill a form to request a VM. After a few emails (and days because time zones), you got a few years old RHEL VM without root access, but also without any real support from the ops organization. They tried to provide the generic cloud experience with VMs and failed.
Ten years ago I loved the cloud because it broke the monopoly / organizational dictatorship of IT in large organizations, always a wellspring of "daily wtfs".
But now the cloud is becoming too powerful, and too small in number. We NEED a competitive-ish on-prems approach... but... what is it? Does anyone have a template to do so? They aren't well advertised that's for sure.
AWS is NOT CHEAP. It may be slightly cheaper than the old monolith IT orgs' DCs, and far more flexible, but it's not cheap. There has to be a way to accept less flexibility for substantial savings over reserved instances and negotiated discounts.
For christsakes, a Raspberry Pi would make a perfectly good server for massive ranges of use cases. 20$ and ultra low power.
Or at least, that's part of AWS' sales pitch. But it makes sense, if you're just an application developer, you can fire up a hosted, scalable, redundant cluster fairly easily and quickly, whereas if you go self-hosted you need to buy hardware, colocation, and either know or hire people that can set it up and mange it.
You just might impress top brass and convince them to get on this bandwagon before they are disrupted.
But yes technically it's "cloud" in the sense that it's still someone else's computer for rent.
Scale does NOT matter for the vast majority of usecases. For example, the parent comment talked about on-prem exchange server.
Working in enterprise software, I was am always surprised when I find a killer feature may not the functional value of something we've built, but that fact it works around some internal disfunction of operations between internal teams.
Most businesses will look at the relative costs and opt for a private cloud solution, then price in the risk and go for the cloud solution. There is no reason the Guardian must control its own hardware.
No, but it has potential political ramifications, right?
All the Cloud providers are US. Does this mean the guardian will be more wary about saying anything negative about the US?
All the cloud providers are big companies, does this mean that the Guardian might be less critical of big companies?
Obviously being completely vendor agnostic is difficult; but we shouldn't pretend that there could not be backlash in unexpected ways- even if "hosting is not a core competency".
A thing that comes to my mind is Edward Snowden's revaluations: given the things he talked about (NSA was even tapping Google lines, for example): how could any whistleblower expect to not be exposed by accident if everyone is using a cloud provider in the US?
It just strikes me as marginally naive to assume that nothing can/could go wrong here, you hand off an immeasurably large amount of control to an ambivalent entity and whatever regime they happen to be under.
I now realize the cloud providers are willing and able to express political opinions and delist people/companies that disagree with political parties.
We're betting that not everyone will be in the main public clouds for these reasons and prepared for the pendulum to swing back for some applications. Financial services companies for example still have major on-prem / self-managed footprints. They are very sophisticated consumers.
(My company Altinity was the one cited at the head of the article.)
By that logic a bookseller has no business running the world’s largest cloud!
Not that I’m disagreeing with you in general, Amazon is very much an exception to the rule.
You can do nearly everything "the computing cloud" offers without it : just use servers, install your services on it, manage it, invest in more servers/ more powerfull servers if needed... Cloud computing is not a brand new invention, it's just delegating the server and services management to others... Whereas, a century ago, electricity was actually an essential input.
Then I can imagine SAAS companies offering discounts if your API requests originate from AWS cloud.
Maybe even Amazon prime members will get free bandwidth from their home internet connections into and out of AWS.
At some point, the whole lot becomes a walled garden, and being outside it is more expensive if you need to use services that are inside.
Although this idea makes practical sense, from a financial standpoint I don't think there will be enough incentive.
Bandwidth savings in infra will represent <1% savings relative to the SaaS/API price.
... and have the staff with experience and resources to keep them running. These days, UPS maintenance is sourced out to the cheapest bidder and testing is infrequent, with the predictable result of shit hitting the fan at the exactly worst time.
Edit: Also I was way more fit from climbing around a building all day cabling stuff. Genuine question - does anyone know what sort of IT jobs require physical work? I miss those days a lot!
I have seen a few positions in the US for datacenter technicians. Those positions had explicit requirements(probably because of ADA) to be able to lift over X lbs as part of day to day activities.
I don't think it was ever profitable unless your energy prices can compete with what industrial-scale miners get in favourable locations (right next to a power plant in an area with too much surplus power, etc).
"Is it really profitable to get in, like now?" Yes, but... Long version: Yes, you will make your money back within 2 years almost guaranteed on a new mining rig. There is a minimum cost to get in. For example, at a minimum, you will need a good card at 3k+, which puts you at 3.5-4k just for price of entry. However, you will make that back within a year.
Now the heavy BUT:
1. No one knows what will happen once ETH switches from mining to proof of stake in 2022. Mining profits may go to 0.
2. If BTC price goes to hell, so do your profits in the long run. This is price-risk, regulatory-risk, etc.
3. There are heavy tax implications and a lot of management. For example, you won't get the returns mentioned if you don't properly account for equipment costs.
4. Once you want to make more than $10 a day, there are heavy, heavy set up costs - cooling, proper power supply, etc.
Finally, profitability of mining is back because China is actively banning crypto mining, so other areas can compete now.
"I don't think it was ever profitable unless your energy prices can compete with what industrial-scale miners get in favourable locations (right next to a power plant in an area with too much surplus power, etc"
This is simply not true Nextgrid, see https://whattomine.com/ for roughly-valid numbers. Note that BTC is not anywhere on the GPU list - you do need extremely expensive ASICs to mine BTC. You also need to order them from China, pay a 25% import tax, and overall risk quite a bit.
Final note - if you could get GPUs at cost or MSRP, you would be in pure profit in less than a year. The problem is that a $2000 GPU is marked up to about $3500 atm.
anyways, is $15/Mh a good buy because that is what it seems to be
It won’t make you rich, but it will keep you busy and get you meeting people in your area.
Our website is https://gardunha.net
Feel free to get in contact if you want to chat more.
Major vendors like Amazon and Google are happy to provide this to anyone and with simple on-boarding, but, as is clear in the article, it's expensive. With Unikraft[0], you can completely re-package your application as a unikernel, automatically. It's a single VM image and does nothing more than just running the application. It boots straight to the main application, it doesn't have a shell, and has nothing else running along side it. The result is a huge cost saving (up to 50%), since you're no longer having to pay for these extra resources just to run the a full traditional OS image plus your application.
[0]: https://unikraft.io
Those sort of numbers without even having a pricing page give me 0 confidence sorry.
One example of recieving a 50% cost saving is from Amazon Machine Images (AMIs). We compared the cheapest official NGINX AMI available on the Amazon Marketplace[0] and then also ran an NGINX+Unikraft AMI. We ran the same workload using wrk[1] and then checked the bill at the end of the month, roughly $80 vs $40.
[0]: https://aws.amazon.com/marketplace/pp/prodview-xogyq23b3mfge
[1]: https://github.com/wg/wrk
Most of the costs for using the NGINX AMI + t2.nano come from software (0.29/hr), not EC2 (0.006/hr).
That's like the worst instance you could pick to prove anything about the performance of your platform. At most you're telling me your AMI costs half of NGINX's... Which still is a lot more than just running Ubuntu with a free AMI.
It’s outsourcing the blame. This is the key to boardroom success. When something goes wrong in their own data centers, the CXOs need to shoulder the blame.
Large corporations outsource everything, including the guy who cleans the copiers, copier not working, blame the maintenance vendor.
Something went awry with the infrastructure, blame AWS
App development project went down the tubes? blame the contractors & outsourcing companies.
The Cxo comes out blameless each time
the problem with applying this concept here is the hyperscalers are not exactly a counterparty likely to underprice IT risks. in any case is there any public evidence of this "blame transfer" in action (pressumably with associated remedies)
There must be a pressure by competition to keep this incentive in check.
It shouldn't be called a "cloud" in that case. It should be called a "fog".
The key challenges in this space are standardization of commodity features, making the whole business of using any of these solutions less complex, and driving cost down. This goes against how the big three are making of course which is exactly through vendor locking aggravated by complexity. Once you figure out how to use this stuff, you have a lot of non portable stuff depending on that vendor. Replacing that costs money and is never a priority. That's why AWS stuff can be such a PITA to deal with. It actually ensures customers can't move away that easily.
However, the common feature set shared by most of these cloud hosting solutions (running vms, databases, docker, load balancers, networking, etc.) is a combination of expensive to use and still too hard to manage. So, the game for new players in the market is being more convenient and less expensive to use. Ultimately private cloud and the cost of getting started as a PAAS provider getting lower means that this is a race to the bottom. Most of the smaller providers already lean heavily on OSS solutions.
The question is whether the big three can long term afford to not support the emerging standard stuff. For example at least for me all the convoluted proprietary AWS stuff is actually increasingly a reason for me to avoid depending on them. It was great when they were the only company with decent solutions but that is increasingly less true.
Yes if you only use compute and storage moving is easy.
If you want scaling in AWS then move to Azure I think that would involve a lot of changes (as it would the other way) I am not sure how much of it Terraform can translate.
I would guess K8 can do this is a vendor neutral way but I am not sure since I have never worked with it.
I do know for a fact that getting your devops pipeline set up in AWS is really complicated. We ended up having to hire two "elite" AWS consultants for 14 days to get it all working. I was not involved directly in it, but my team had great need of the finished pipeline. If we went with Azure we would have to do it all over again. But most of our stuff is in .Net so getting it all working o Azure would have been magnitudes better.
The more "premium"/ high level services one starts using the ability to move on prem or switch cloud becomes harder and harder. Which is of course the goal for the cloud vendors.
I have been in companies doing both.
It is my experience that it is a lot easier on Azure than AWS. (At least as AWS was 2 years ago and Azure one year ago).
With the new overlords, I have to pass through fewer reviews (I work in financial sector; we love review meetings and gates.) and provisioning a service is as simple as a AWS/GCP/Azure url. And tearing down stuff is easier. The old overlords of IT would be pissed if we decided we did not need a h/w server and we would have to fill out questionaires to determine how we could have been so misguided and then I had to wear my dunce hat for any meeting we would have for the next requisition.
I love the battle of cloud computing. With the big three, they are still hold too much power. I look forward to the day when Vultr, DigitalOcean, Ovh, Linode can get approved to be used in my employer's IT stack. With Terraform or Kubernetes, I would be able change our dev/qa/uat cloud environments at will and this will force the Big Three to lower their prices even more.
In new cloud land you're essentially contracting out a team at Amazon to handle all that. You can rent a team that is dedicated to that one particular service. In cloud land behind every service there is a whole team of people dedicated to maintaining that particular service, in on-prem land behind every service is a 1/20th of the capacity of your IT team.
Lot harder to refund that new server that was just installed on the rack.
This stuff has nothing to do with cloud but mainly on how the business thinks off and controls IT.
Some finance businesses have the resourses to hand to compete with the cloud providers or at least create internal self managed equivalent
what does this mean
requests != bandwidth. are they paying a penny per request?! that's a lot. or is their website several gigabytes? or are indian CDNs expensive?
highly suspicious of economic news service that doesn't understand the numbers it prints