Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?
1) internal users: mainly developers by providing CI/CD
2) external users: end users
The platform of choice is mostly Kubernetes these days, which promises among other things stuff like
- load balancing
- self-healing
- rollbacks/rollouts
- config management
In particular, a main critic is "state" and the possibility to change that state by e.g. messing with config files via SSH, which makes running and maintaining these servers more error-prone.
However, my main question is:
"If this old way of doing things is so error-prone, and it's easier to use declarative solutions like Kubernetes, why does the solution seem to need sooo much work that the role of DevOps seems to dominate IT related job boards? Shouldn't Kubernetes reduce the workload and need less men power?"
Don't get me wrong, the old way does indeed look messy, I am just wondering why there is a need for so much dev ops nowadays ...
Thanks for your answers.
432 comments
[ 1.8 ms ] story [ 398 ms ] threadThe number of single-server setups with kubernetes thrown in for added complexity and buzzwords I’ve found is way too dang high.
Kubernetes helps with: networking setup, consistent deployments, task distribution. That's about it. It's more standardised than plain VMs, but you still have to deal with the other 90% of work.
Because we're living in the stone age of DevOps. Feedback cycles take ages, languages are not typed and error prone, pipelines cannot be tested locally, and the field is evolving rapidly like FE javascript did for many years. Also I have a suspicion that the mindset of the average DevOps person has some resistance to actually using code, instead of yaml monstrosities.
There is light at the tunnel though:
- Pulumi (Terraform but with Code)
- dagger.io (modern CI/CD pipelines)
Or maybe the future is something like ReplIt, where you don't have to care about any of that stuff (AWS Lambdas suck btw).
Ironically, "DevOps" started as a philosophy that developers should be able to do operations, e.g. deploy, monitor their apps without relying on external people (previously called Sys Admins, etc). Yet, we're at a stage where the "DevOps" role has become the most prevalent one. IMO things have temporarily gotten slightly worse to get much better later.
From the productivity standpoint, it is not acceptable that a Machine Learning engineer or a Full Stack Developer are expected to know Kubernetes. Or that they need to interact with a Kubernetes person/team. It is an obstacle for them to produce value.
Kubernetes is not THE solution. It's just an intermediate step. IMO, in the long run there'll be very few people actually working with technologies like Kubernetes. They'll be building other, simpler tooling on top of it, to be used by developers.
You already named few examples. I can name few more:
I agree - these things should be abstracted from the developer - thats the goal of SRE/platform engineering - DevOps is [supposed to be] as you said, a philosophical and cultural stance around early productionization. While not mutually exclusive, they're not the same thing.
But back to your point re: orchestration-level concerns being foisted upon devs - at a shop of any size, there will be devs who feel they _need_ to touch kubernetes to get their job done (wrongly, IMHO) as well as devs who want nothing to do with it - so without engineering leadership throwing their support heavily behind a specific approach, its hard for a small team to deliver value.
Hard pass.
That's valuable because, on the scale of large companies, it's much easier to hire "a network expert" or "a storage expert" or even "a Gatekeeper policy writing expert" than to hire a jack of all trades that can do all of these things reasonably well.
The corollary from this observation is that Kubernetes makes much less sense when you're operating at a start-up scale where you need jacks of all trades anyway. If you have a team of, say, 5 people doing everything from OS level to database to web application at once, you won't gain much from the abstractions that Kubernetes introduces, and the little that you gain will probably be outweighed by the cost of the complexities that lurk behind these abstractions.
Traditional deployment models also have ways to separate the network, from the security, from the storage, from the application stack. But it's typically done through strict change control processes.
I see many orgs choose to change to K8s because it offers improvements to operational tasks related to provisioning all of those changes, and speeds up the old change control gateways.
However, K8s is tuned to operate extremely profitably in organisations that need to make large numbers of changes to their infrastructure or software stack all of the time. And begins to break down at scale (at least from a cost point of view) compared to alternative solutions if organisations are not meeting minimum requirements for deployments a day.
IT orgs have chosen to adopt this huge piece of operations software that is itself fairly monolithic, and requires large amounts of upkeep and maintenance to keep it running smoothly and provide constant availability.
But even though they've adopted this new massive fixed cost in their IT operations. They continue to use old change control processes, often because restructuring old teams, that have traditionally made the company a lot of money, into new teams proves to be an incredibly risky exercise.
And so, the net outcome, is that their overall IT operations processes marginally improve at best. But they now have simply absorbed a new fixed cost on top of all their existing ones. What's worse, I see in some orgs, the additional cost pressure is being noticed, but they attribute (IMHO incorrectly) this cost pressure to lack of competence in the new technology (K8s), and begin a massive hiring spree to try and find specialists to better tune the technology stack. The solution, IMHO, instead should be to push hard on existing teams to simplify and downsize, and create incentives to interact with the infrastructure APIs more aggressively, letting Sysadmins/DevOps/SREs deal with faults, errors and failures the way they always have, but with the new fancy tools that let them work more efficiently.
SRE is a specialized software engineering role -- you'd hire SREs if you wanted to create something like Kubernetes in-house, or do extensive customization of an existing solution. If you hire an SRE to do sysadmin work, they'll be bored and you'll be drastically overpaying.
DevOps is the idea that there shouldn't be separate "dev" and "ops" organizations, but instead that operational load of running in-house software should be borne primarily by the developers of that software. DevOps can be considered in the same category as Scrum or Agile, a way of organizing the distribution and prioritization of tasks between members of an engineering org.
---
With this in mind, the question could be reframed as: if projects such as Kubernetes are changing the nature of sysadmin work, why has that caused more sysadmin jobs to exist?
I think a general answer is that it's reduced the cost associated with running distributed software, so there are more niches where hiring someone to babysit a few hundred VMs is profitable compared to a team of mainframe operators.
This makes a lot of sense. The same thing happened in the past with new technology, such as the electronic spreadsheet:
"since 1980, right around the time the electronic spreadsheet came out, 400,000 bookkeeping and accounting clerk jobs have gone away. But 600,000 accounting jobs have been added."
Episode 606: Spreadsheets!, May 17, 2017, Planet Money
[1] https://fred.stlouisfed.org/series/CLF16OV
"Though the spreadsheet was supposed to make clerks obsolete, it in fact just upgraded their job requirements to allow the operation of spreadsheets"
And probably their salaries too.
This is more in line with the discussion related to OP, as well, as devops employees exist, even though apparently it is automated.
Given that the US has transitioned out of manufacturing and into businesses services - I don't think much of this is explained by technology creating new jobs.
I think it's just explained by the workforce growing - and the shift in the US's role in the global economy.
So the real question is - whether the ratio of different roles changed vs total population.
Go back and read a few DevOps books and blogs by the founders of it. We will always need separate disciplines for dev and ops, just like we need mechanical engineers and mechanics/racecar drivers. But we need them to work together and communicate better to solve problems better and not throw dead cats over walls.
You can of course give devs more powerful tools, more access and agency to enable them to develop the software better. Sentry.io is a great example of what is needed; makes everyone's life easier, devs can diagnose issues and fix bugs quickly without anyone in their way. That doesn't require operations work because it's just simplifying and speeding up the triage and debug and fix and test phases. It's the fundamental point of DevOps.
You want the person who designs the car to know what a car is, and to be able to diagnose basic issues like "the fuel gauge says 'empty' and engine won't start". And there's no analogy to an Indy car driver in software, every distributed system is self-driving.
The most popular alternative to "DevOps" is a team of developers who do not run the software, and may not even have the skills or capabilities needed to boot up the server process. They do their development in an IDE, run unit tests to verify functionality, and do not have permission to log in to the production environment.
Meanwhile the "ops" side consists of people who may know basic shell scripting, or some Python if they're a go-getter, but are unable to read stack traces or diagnose performance excursions.
---
None of those have any relationship to DevOps. I was on Borg-SRE. My job was to build the tools and maintain the cloud infra. The point of having an SRE team is to avoid having a "small army" of ops people. Manual operation of compute infrastructure is uneconomical at scale. Google does expect its developers to understand basic principles of distributed systems, yes. What I mean is that when your service runs 24/7 and downtime is reported in the New York Times, there is no room for manual action during normal operation. The system must have the capacity to maintain itself and self-recover.As an outsider, that command looks really easy to mess up.
Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right. This will light up warnings if you get anything wrong;You get:
So structurally,1) in the code realm, for doing minor work we have really strong guarantees,
2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker.
I think this is behind the feeling that the k8s system is not ready yet.
>> None of those have any relationship to DevOps.
Deming. I think you mean maybe that Deming has less of a relationship to SRE (Google style). The DevOps Cafe guy wears a Deming t-shirt and talks about Deming all the time. Maybe he has denounced Deming for some reason?
Value stream mapping. This has been brought up in at least one talk at DevOps Days.
Six Sigma. There's some obvious overlaps if you look it up.
Cool, you probably know my old colleague Hugo, he was one of the first SRE's at Google in a team of like 20 or so. (I wasn't there, could be mistaken).
Anyway, what "Production" was, is different than what devops is. DevOps is different things to different people. In the beginning it was "What happens when systems administrators do Agile".
(Genuinely), Confusion came about because of the "100 deploys a day" talk and the conference being called "dev ops days" (to include developers).
The two things got conflated and now everyone thinks devops are.. well, build engineers or sysadmins or developers who learned a bit of bash and terraform.
From what I have seen, it's also getting familiar with the inner workings and process of the company, as well as current and past runtime tendencies for performance and uptime.
That sounds like sysadmins.
DevOps is how Engineering respects the work of sysadmins.
You typically build systems to sell a product. The driver is the customer that uses the app. You build the app to be as simple as possible to effectively operate while providing the best possible outcome for that customer. The driver doesn't care or need to know how the engine or the rest of the car works, just that the engine and car works well.
- each role in a company tries to optimize/nudge whole organization toward this role's convenience.
- specialization improves local optimum (advances certain role) at the cost of global optimum (everybody has to dance around new roles processes)
- joining seceral roles into one, creates the oposite result, optimum is searched at more global level (not necessarily found)
- Separation of responsibilities (aka creation of new role) can generate a fractal (e.g. tester of left winglets blue stipe's thickness meter)
- complete joining off roles will create homogenous chaos after n employees (everybody should do everything)
prediction: we will see constant experimentation, first roles will be split, then some will get joined. then split again. then joined again. (people cant search for local optimum and global optimum at the same time)
Lesson learned: kubernetes isn't that bad
I agree with this definition of DevOps. However the vast, vast, vaaaast majority of real life uses of the term "DevOps" I've seen are just rebranded sysadmins. Sometimes it at least implies a more engineering approach to their coding. But in these institutions the Devs and Ops are very much separate groups of people, unfortunately.
If you're lucky the hiring manager might have at least read the Phoenix Project or the Google SRE books.
1) Kubernetes is an infra platform for the ops in DevOps. If developers need to spend a lot of time doing Kubernetes it takes away from their ability/time to do their dev. So, there are a lot of platform teams who pull together tools to simplify the experience for devs or DevOps specialists who handle operating the workloads.
2) Kuberentes is, as Kelsey Hightower puts it, a platform to build platforms. You need DevOps/SREs to do that.
3) Kubernetes is hard. The API is huge and it's complex. The docs are limited.
The biggest challenge with learning Kubernetes is that third-party documentation is largely funded by ecosystem startups flogging their weird product. It can be very difficult to figure out things like network overlays when there's so few examples that don't involve third-party tools.
At the end of transitive reduction, you end up with a graph with all redundant dependencies removed but functionally, it is still the same graph.
[1] https://en.wikipedia.org/wiki/Transitive_reduction
This is wrong, deploying on Kubernetes is easy and quick for most apps, you have one docker image one deployment spec and that's it.
https://kubernetes.io/docs/concepts/workloads/controllers/de...
Probably 98% of the devs were blissfully unaware of that complexity that the charts abstracted, and it let them focus on the services they were writing. I wasn't one of them, and always made sure to thank the devops team for simplifying the day to day deployments whenever I had to deal with writing a custom one.
It Handles the devops jobs for dev teams.
Full disclosure, I work for Bunnyshell.
Bunnyshell makes it easy to create and manage environments. (EaaS - environments as a service)
You connect your k8s cluster(s) and git accounts/repos, it reads the docker-compose files and creates deployments on the cluster.
You don’t need to know or write Kubernetes manifests, those are created for you.
You also get auto updates and ephemeral/preview environments (when a PR is created against the branch of your env, Bunnyshell deploys a new env with the proposed changes).
You are not restricted to creating resources only on the cluster, you can use Terraform for any resource that is external to the cluster ( like S3 buckets, RDS instances, anything Terraform can handle).
Hope this helps,
A day's work for one person to accomplish that isn't bad, IMO, but what that doesn't capture is the literal weeks I spent poring over documentation, trying things, running tests, learning what didn't work (Rook + Ceph is a nightmare), and so on. I went so far the day before the cutover as to recreate my homelab in Digital Ocean and run through the entire installation process.
Having services that magically work is hard. Having a golden path so you can create a new one with a few clicks is even harder.
There's a sort of common idea that a "Kubernetes cluster" is an entire self-contained PaaS, and that (for example) monitoring in Kubernetes is somehow fundamentally different from what came before. It's easy to fall into the trap of creating an internal clone of Heroku, but Kubernetes itself doesn't require you to do so and it can be a lot faster to just run Nagios (etc).
Well, load balancing is part of Kubernetes (except that it does not provide a sane implementation), and tbh having to debug strange failures caused by seemingly-innocent Service configuration is my least-favorite part of Kubernetes.
I agree with you on other points.
I’ve seen actually decent engineers (maybe they’re not decent?) bring down prod because they accidentally Kubectl deploy’ed from their command line.
Also, self-healing can create interesting problems which are fun to trace, debug and understand.
Sure could, if it provides benefits that outweight the additional work. Insurance companies will happily pay a couple more IT-specialists if it reduces the amount of cases they have to cover by an arbitrary percentage. Tools very much can serve other purposes than reducing the friction of human attention.
If you can containerize your infra and thereby mitigate threat vectors that may put your entire enterprise at risk if exploited, that's a good business call to make, even if the cost of your IT-department gets inflated by some margin.
Does it add work compared to setting up a VM with Docker and Watchtower? - for sure..
But does it add work compared to setting up something that gives you all the same benefits of k8s without using k8s? - imho definitely not.
That's the catch, if running your app on a manually setup VM is equivalent to running your app on Kubernetes then you don't understand what Kuberentes is or provide.
More work means
- more employees needed
- more direct reports for managers
- more jobs for x technology
- more people learning x technology because of those jobs
- more StackOverflow questions for x
- more companies using x technology because it's "easy to hire for"
- more money to be made in teaching x
- more learning material for x
https://www-users.cs.york.ac.uk/susan/joke/cpp.htm
Warning to bashers...it's a fake interview...
Kubernetes is a great tool if you have scaling management problems.
"Things" here mean different things for different people. For me, it's secrets/certs, better rolling deploys and infra as code.
I agree with some other comments in this thread about a general fervor in the Enterprise space to "modernize" needlessly. This conversation usually lands on the company copying what everyone else is doing or what Gartner tells them to do. Cue "DevOps".
100 percent agree with your comments on something simpler. I can't tell you how many times I've debated with our Analytics teams to just use Docker Compose/Swarm.
I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff.
Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.
Once we deployed, we ran into all sorts of SRE-issues. Turns out AWS sets all those “sane limits” that our own folks never did. Still hunting ghosts from the rollout 6 months ago.
Makes for good resume fodder, and makes me laugh at the prestigious titles and positions the folks who built this system went on to receive at big name firms.
Guess it is someone else’s management problem now. :shrug:
For what it's worth, deploying in Kubernetes is pretty easy once you figure it out (and often finding the information is the hardest part). All you need to do is update the Deployment resource's "image" parameter. You can do that with `kubectl patch` like so:
Kubernetes will handle bringing up a new replicaset, validating health checks, draining the old replicaset, etc.If everything is in code, and you have a reasonable branching strategy, it's much easier to control change, to rollback bad merges, to run pre-hooks like security checks and configuration validation tools, etc.
I think you're assuming that invoking kubectl means invoking it directly from a user's command line, but kubectl can also be called in a CD script.
This state can be version controlled, written in stone, whatever you prefer and it can always be attained.
When administrators start issuing imperative commands to a cluster, state starts being changed and there is no record[0] of the state Kubernetes is being asked to maintain.
[0] Not entirely true, the state can always be retrieved from the cluster so long as it hasn't failed.
Firstly as you point out, there is a record of the state Kubernetes is being asked to maintain: it's in the API server as the spec of each resource.
Secondly, using `kubectl` "patch" in the manner described is not making changes to the cluster state directly, it's making changes to the specification of what should be maintained, and the various controllers effect the state changes.
Fundamentally, the argument seems to come down to "you don't have a record of what you once asked the API server to do", and that's fair enough - you don't. But that has nothing to do with imperative or declarative models.
I'm not advocating actually doing this on a day-to-day basis, but the arguments against it are not ones of imperative vs declarative.
[0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...
Applying a patch is "make these changes to your existing state".
That dependency on existing state is a difference, and it seems to map reasonably well to what declarative/imperative seem to usually be used to mean in this context.
2. "make these changes to your existing state" is still declarative
It isn’t though. It’s “please make the state look like this eventually”. You do not patch state, just spec. Controllers effect the changes.
That said, the parent is wrong that a full manifest is declaring the (etcd) state in a way that a (strategic merge) patch isn't--both are declaring etcd state, but a strategic merge patch is doing so in finer-grained increments. A strategic merge patch can declare zero state or many full manifests, while applying full manifests can only work in increments of complete resource manifests. But both are telling Kubernetes "this is your (etcd) state now".
You do actually have this record. First of all, because k8s has an audit log, and secondly because deployments maintain a revision history (so you can always rollback--kubectl even supports this via `kubectl rollback undo`).
Updating the manifests, pushing them to version control and having CD deploy them encourages humans to "do the right thing".
It would be quite easy though to just tweak that one environment variable though while I patch in the new image version to update that one service; until the entire cluster dies and I can't retrieve the last definition and need an equivalent cluster up within the hour.
This is really more about the practice of writing down the state you want the cluster to be in (the spec) and showing it to the cluster, than just ordering it to do one thing without context.
In this sense, Declarative Vs Imperative is just a proxy term for "do I have a record of the state I asked the cluster to keep?"
Moreover, people in this thread also don't know what "declarative" means. The patch is declarative, and "declarative" doesn't provide the claimed benefits. For example, as with the patch command, I can create, apply, and delete a Kubernetes manifest file (indeed, I can apply directly from stdin!) and there is no additional record of the change beyond that which would exist if I had just applied the patch.
I will grant you that `kubectl delete -f` breaks my argument, since it's imperatively modifying objects given a declarative manifest. Re: changes via stdin, I mean sure; you can also pipe a deployment manifest into `ex` or `sed` to make changes on the fly and then echo it back to `kubectl apply -` but I wouldn't recommend it.
Also, working in raw manifests isn't a panacea; if you delete a resource's manifest, Kubernetes won't delete its corresponding resource (maybe this is what you were referring to with your bit about `kubectl delete -f`). You need something that diffs the manifests against the current state of the world and applies the requisite patches (something like Terraform) but this isn't related to how you change image versions.
[0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...
I managed my stateful sets/services for a while with terraform, but my experience was absolutely terrible and I have stopped that eventually. I now use "kubectl patch" and "kubectl apply" with handwritten yaml, but the workflow feels very clunky.
Intuitively it seems obvious to me that there must be a tool helping with that, but for some reason I was absolutely not able to find anything that would be even a little bit helpful. I am considering writing a couple python scripts that will automate it.
If you need something more flexible than that, try making your own helm chart. Helm will give you some text templating, pre and post hooks, some basic functions, and some versioning and rollback functionality.
You can start simple by just pasting in your existing k8s yaml, and then just pull out the pieces you need into variables in your values file. If you need to change an image version, then you just update the variable and `helm upgrade mychart ./mychart`
I only use ‘kubectl apply’ for small stuff where I only have a couple resources. Anything more complicated and a tool like helm is much more useful.
* shell interactions with quotes * double quotes * interpolating into image namespace with no autocomplete * easy to forget an argument * do you get autocomplete against the deployment name?
Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right.
* var x = new List<Dictionary<CustomType,int?>>()
This will light up warnings if you get anything wrong;
You get: * go to definition of `CustomType` * autocomplete on classnames * highlighting on type mismatches * warnings about unused vars. * if you initialize the var, the IDE will try hard to not let you do it wrong
So structurally,
1) in the code realm, for doing minor work we have really strong guarantees,
2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker.
I think this is behind the feeling that the k8s system is not ready yet.
2. kubernetes will abort the change if it doesn’t syntax check or the types don’t match.
> I think this is behind the feeling that the k8s system is not ready yet.
Whether you patch the deployment image from bash or from C# doesn't indict the k8s ecosystem.
or
https://www.shellcheck.net/
What I'm trying to say is that as an operating system, Kubernetes is now a young adult, and historically speaking, operating systems at this level of maturity have been adopted and ridden for decades, with much success. But, ya know, if you chose OS/360 in 1996 you would have a point.
This a stretch, but to me Kubernetes is like the C programming language for infra. If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel). I assume any commercial (or non-!) "higher-level abstraction" for infra is already (or will be soon) built on top of Kubernetes. I am OK with it.
I write this post as someone who is uber-techincal, but I know nothing about actually using Kubernetes. I can do vanilla "hand-coded/snowflake" infra just fine in my constrained, private cloud environments, but nothing that scales like Kubernetes.
I might be nitpicking, but I'm not sure that's necessarily true. You could in theory write a compiler for a new language in C (or even assembly!), and once you have a working language, re-write the compiler in that new language. Now that there is no C code involved in the stack anymore, would that still count as a C-turtle?
Haskell for example, has some "bits" written in C, but a lot is written in Haskell or assembly[1]. So if you look at the WHOLE stack you'll find C _somewhere_ (probably most places), but I don't think _everything_ boils down to C.
Granted, a LOT of stuff is written on top of C.
[1] https://stackoverflow.com/questions/23912375/what-language-i...
After seeing so much great work done with JavaScript in the form of "transpilers", I think a lot can be done in that area. I feel Zig is a crazy good idea: A brand new language that produces binaries that are 100% compatible with C linkers. If all goes well, in a few years, why would anyone use C over Zig? It seems like the future.
Lots of people think C++ is bat sh-t crazy complex (me too, but I still like the mental gymnastics!). What if there were different C++ dialects supported by transpilers that intentionally restricted features? I think kernel and embedded programmers would the primary audience.
https://www.oreilly.com/library/view/kubernetes-in-action/97...
If you try digital ocean k8s offering it's fairly straightforward.
Google cloud was the first to offer a decent k8s as a service, if I recall correctly. We didn't have DevOps back in 2015 and we were on GCloud.
Personally I don't pick k8s just because it's heavy to run and I don't want to waste machines (plain docker is good enough for a large part of what people actually need). Sometimes in a project when I can't figure something out with just docker, I just bite the bullet and install k0s.
What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes.
The core system is great, but we need to replace these DevOps with a framework or platform.
Doesn't Rancher fit this description? It's pretty resource-heavy though.
- Deploys kubernetes clusters - includes Fleet for CI/CD - install apps with helm - ISTIO (awesome)
It feels like it is positioned for orgs with large needs. I'm looking for k8s for small nimble orgs.
I will look at Rancher more, thanks for reminding me of it!
Google's Anthos is also hard to describe now as they cover similar "everything" product features.
GCP now has Autopilot which lets you pay just for the cpu you use (no cluster management at all).
Anthos includes ISTIO which may at some point work on Autopilot. This would mean not having to fiddle with GKE Ingress (which I found unpleasant) and instead use the new standard Gateway.
I believe that eventually GKE Autopilot will offer running individual pods on GPU/TPU, pay as you go.
But when is this all as easy as using Next.js ?
YAML: significant whitespace is always unwelcome but YAML also introduces unexpected problems, like how it deals with booleans. For example, say you have this array:
You might think this is an array of strings, but you'd be wrong.It's also difficult to read through a YAML config and understand the parent of each key, especially in code reviews or just on GH.
I believe k8s life would have been easier with JSON configs, where it's impossible to confuse e.g. booleans for strings and where it's easier to understand the object's hierarchy.
Helm's use of gotpl: this choice exacerbates the problems with YAML. Now you're treating a structured language with a text template library. You have to spend energy thinking about indentation levels and how the values will be interpreted by both the templater and k8s.
I think helm would be less frustrating if they chose some templating library that made objects first class citizens. Where you can inject values at specific locations with one-liners or simple blocks of code (e.g. `ingress.spec.rules[0].append(yadda yadda)`)
I'm sure there was debate about these choices early on and I don't have any unique ideas here, so I don't want to be too critical. These are just a couple of pain points I've personally experienced.
["x","y","z"]
["x", True, "z"]
[1] https://yaml-online-parser.appspot.com/?yaml=foo%3A+%0A-+x%0...
If you're going to use JSON for config, it's better to use an actual JSON parser.
https://john-millikin.com/json-is-not-a-yaml-subset https://news.ycombinator.com/item?id=31406473
You could mitigate some of the issues and get JSONs "strictness as a feature" by passing the document through e.g. `jq . $file` as a CI step, but I don't think that'd resolve the 1e2 issue. TBH I didn't know you could write numbers in JSON like that, so I imagine it'd be an issue that doesn't come up often. But it's disappointing that it wouldn't just work.
That's false. http://p3rl.org/JSON::XS#JSON-and-YAML
Technically YAML is (supposed to be) a superset of JSON - (almost) all valid JSON is valid YAML. So you could write all your configs in JSON and they'd (probably) work just fine (assuming you keep things relatively simple).
My cofounders and I are working on a solution to this at https://www.jetpack.io/. If you're interested in early access, we'd love your feedback!
eksctl create cluster --name mycluster --region us-west1 --with-oidc --fargate --external-dns-access --asg-access --full-ecr-access --alb-ingress-access
Now the fun starts:"Kubernetes Failure Stories"
https://k8s.af/
Sure, one can break anything, but our anecdotal experience is we’re now focused on actually delivering code rather than fretting about node failures.
https://news.ycombinator.com/item?id=31581372
Definitely a lot of work.
...until you need to debug something somewhere in the enormous stack.
The lower level infrastructure/platform/kubeadm type stuff isn't really 'Dev' related at all.
Kubernetes only exists, because Google lost the Cloud Wars, and this was their Hail Mary pass.
At my first job outside of Google we used something called Convox, which was very similar to running things in production at Google. You triggered a package build from your workstation (!) and then adjusted production to pick that up. Very similar to mpm packages and GCL files. (The difference is that Google had some machinery to say "while the build might have been triggered from a workstation, all this code has actually been checked in and reviewed". Convox did not have that part, so yeah, you could just edit some files and push them to production without checking them in, which wasn't great. But when you are a 4 person development team, not the end of the world by any means.)
All this is not to say that Kubernetes is the right solution to your problem, or to almost anyone's problem. It was just an improvement to Google's infrastructure, for the sort of problems that Google had. For some people it makes sense... for you it might not.
Borg is far more advanced in its scaling abilities.
If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes" and it would decrease their workload.
In reality this does not seem to be the case, that's why I asked.
I'd say it would result in either:
- the same scalability and availability with fewer DevOps engineers - better scalability and availability with a similar number or more DevOps engineers
In my experience, it's almost always the second case that happens. For example, a service would be moved from a few (virtualized or physical) servers that can only be scaled manually, to a k8s cluster with either autoscaling or at least scaling by changing a configuration file.
But what would you suggest in lieu of Kubernetes? What would save work for a shop which is not yet a giant but has already overgrown the capabilities of 2-3 manually managed boxes / instances?
I can think of several options. Management by Ansible / Saltstack / Chef can easily become a rabbit hole comparable to maintaining K8s's YAML. In your experience, does Nomad save SRE work at smaller scales? Does Terraform? CloudFormation?
It's "Amazon ECS", and it works pretty well for standard fare CRUD web services, but more complex use cases quickly end up pulling in more and more AWS services (for example, if you need to run a cron job, you can use EventBridge to trigger an ECS task or just do it all with Lambda). This isn't dramatically worse--it's mostly just different. Kubernetes APIs are more consistent than in AWS, and Kubernetes tends to be more flexible and less limited than AWS. It's also much easier to extend Kubernetes or find a pre-existing extension than in AWS. But mostly it's not going to make or break your project one way or the other.
If you're just running CRUD web services, this is fine, but if you need to spin up a background job or do more complex orchestration then it can quickly become advantageous to
I run a small cluster for my side projects consisting of 3 nodes. 1 node is dedicated to run database containers and control plane node (it's also a worker node but strictly for database statefulsets only), and the other 2 as worker nodes, but one of them also run an NFS server that mounted as persistent volumes so every worker nodes can have access to it.
Self-healing? Internet-scale? I have on interest for them on my small cluster. I just want my apps to not go down while I'm updating them without writing complicated blue-green deployment scripts, ability to move around pods to other nodes when one got overloaded, and ability to add or remove nodes when needed without starting again from scratch. I basically treat it like docker-compose on steroid. So far it works really well.
This has been the biggest drain on my career. Everyone wants to be an "engineer" ready to handle every problem like its the next facebook. Like bruh, this service is going to get like 100 req/hour max and only when the sun is up - just **ing throw it on cloud run. We can tell the only thing you want to build is your resume.
Pretty much every piece of software I've written in the past decade that tends to have configs in production can also work with little or no config. Including clustered software that just exploits things that are widely available like Zeroconf to get a decent low effort cluster up and running. No, you probably won't be able to (or want to) use those features in production, but that's beside the point. The point is to lower the thresholds. And then keep aiming at lowering them wherever you can because asking other people to care about your software is seriously uncool. Other people will never care as much as you do.
It is normal for programmers to become defensive about software. Be it their own or software they really like. But it is far more productive to assume that when users think something is awkward, perhaps it is because it is awkward. And perhaps it could have been done better.
Nobody actually gives a crap what someone thinks Kubernetes was built for -- and what kind of rubbish experience they think is deeply justified by the goals.
It is either needlessly awkward to use or it isn't. And guess what: most people think it is awkward. And I seriously doubt it needs to be this awkward.
Of course, there's a huge cost to building your own framework as well... And it's easy to get wrong.
I started https://www.jetpack.io/ recently to try and build a better solution on top of Kubernetes. If you're interested in checking it out and giving us early feedback, you can signup on the website, or email us at `demo at jetpack.io`
k8s is not the right tool for every job. Most companies are not at the scale where they need to worry about the problems that it's trying to solve. But it's a cargo cult - they see the blog posts about how moving to k8s solves a bunch of problems that come up as you scale and decide they need to be solving all of those problems also even though there are simpler solutions at their current scale.
There's a bunch of other platforms out there that are way more opinionated and less "awkward" but they don't have the buzz that k8s has.
Kubernetes ought at least to be the right tool for a wider range of compute task. And I think it could have been the right tool for a wider range of compute tasks.
For instance, I don't regard one of our products that runs in three availability zones and has 3-4 instances per AZ as being "large scale". It is still a small system. And it doesn't run in multiple AZs for performance reasons but because we really need high availability.
We embedded discovery, failover and automatic cluster management in the server software itself. But it isn't really how we'd like to do it. But it is still less of a hassle than running K8S. (It also means that we can do that if you license our software to run it on-prem on pretty much most runtime environments, and that has its value, but again, this isn't functionality you want or should have to do yourself)
At ~12 instances probably still in "pets aren't so bad" territory.
I have to manage like half a dozen docker images. K8s seems like a massive overkill, but managing by hand is rather error-prone.
yet its really not great at the first, and I'm strongly suspicious about the latter claim. Its very simple to make boneheaded decisions about networking that makes thing fragile.
In terms of scale, you have a limit of 5k nodes. given how fucking chatty it is, it costs a lot to run loads of nodes.
But DevOps means many things because it's not clearly defined, which also makes it difficult to hire for. It's a "jack-of-all-trades" role that people somehow fell into and decided to do instead of more traditional software engineering.
Also, from what I've experienced from our internship program, CS programs are really bad at covering these fundamentals. Students aren't learning such basics as version control, ci/cd, cloud platforms, linux, etc.
That's because those things are not part of Computer Science, or so I'm told. I got a degree in Software Engineering and regret nothing.
Good. Those aren't 'fundamentals' of 'CS'.
Is that K8s, or that you've outsourced hosting, storage and stateful storage to cloud services?
I suspect that things are much easier to automate effectively, and the knowledge for automating things is much more common.
Now because of new possibilities, and new development they want to switch to Kubernetes to have that new possibilities everyone is talking about, and now you have to build many new containers, configure k8s, autoscalling etc... and developers don't know it (yet) and don't have time to learn it.
So lets hire a DevOps (me) that will do it ;-)
"since 1980, right around the time the electronic spreadsheet came out, 400,000 bookkeeping and accounting clerk jobs have gone away. But 600,000 accounting jobs have been added." Planet Money, May 17, 2017, Episode 606: Spreadsheets!
Read the introduction to the SRE book, available free online [1] - and you'll see that SRE is defined _in contrast to_ systems administration. Its specifically defined as software engineering with the goal of managing operational complexity.
Modern shops' failure to understand this (most SREs haven't read any of the book, let alone stopped to think what SRE actually means) is IMHO a primary factor in the failure of most "devops transformations"
[1] https://sre.google/sre-book/part-I-introduction/
The majority of SREs and sys admins I know don't really want to run Kubernetes. They'll do it, if that's what's called for, but it just adds complexity to trivial problems.
Developers want Kubernetes because it's quick to deploy containers, networks, load balancers, anything they want. That's fair, traditional hosting doesn't have much that provides that level of flexibility or that will allow you to get off the ground at that speed.
The issue is that as a platform Kubernetes isn't that easy to manage. It certain has improve, but it can be difficult to debug when it break. Nobody wants a 3AM page that a Kubernetes cluster is down. We've even seen companies say that they'll just stand up a new cluster if something breaks, because it's faster. Add to that the complexity of components deployed inside your cluster. As an SRE it absolutely suck to get the "hey the network's broken" from a developer, because that means that you now have to be an expert on WHATEVER load balancer or CNI they've decides to pull in.
As great as much of the stuff you can deploy in Kubernetes is, it's still harder to debug than an application running on a VM or bare metal server, with a "real" network.
If a developer wants root in the cluster so they can deploy some strange low-level plugin, then they need to be the ones carrying the pager.
If you need less than 8 instances to do host your product, run far away anytime anyone mentions k8s
There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter.
Forget a whole datacenter, even just one rack is an unimaginable amount of computing power, these days!
For almost all purposes, we don't really need thousands of containers running on unimaginably fast computers, coordinated by AI-driven automation systems. What we need is software that is not morbidly obese.
https://en.wikipedia.org/wiki/History_of_supercomputing
Holy shit, is this considered small to medium enterprise now?
The estimated maximum single cluster is 300,000 containers.
That's pretty medium, I've ran more than a million processes before, and nomad has 1million containers as its challenge https://www.hashicorp.com/c1m
borg can handle this easily.
But I’ve been in situations where it would have been worthwhile. I’ve been in situations with 30,000 machines that needed to be controlled. Splitting them out into very many clusters would be a lot of wasted overhead in configuration, administration and because you lose nodes to masters.
With some exceptions, I believe that's a few orders of magnitude above what a small business can run on. Nowadays people just start their day by drinking some K2l-aid and spinning up a "basic" 6-node cluster for a development prototype.
Maybe I'm wrong, of course.
Is my observation correct that unicorns start to see that scale?
If you want to stay on the Kubernetes route check out k3s. Super easy to setup and usable for small production workloads
I always thought that one of the advantages of going serverless was that you didn't have to worry about keeping the underlying operating system up-to-date. No needing to do a monthly "sudo apt update && apt upgrade" or whatever. But containers throw that all away when container images enter the world.
Instead of updating your operating system, you're updating your images...and it's basically the same thing.
Most would say even a server-ful system (k8s, or whatever) should be considered 'cattle not pets' with immutable nodes replaced as needed anyway. No update, just replace. Just like building a new image and having new pods (or serverless whatevers) pull it.
If you're truly "serverless" by my interpretation of it, then you wouldn't care. Your cloud provider will have updated their infrastructure, and that's all that matters.
In fact almost nothing is serverless (well, the truth comes out! ;)) by that definition, since even Lambda has runtime versions to choose/upgrade, Managed-Acme has Acme versions, etc.
SES, SNS, SQS, etc. sure, but I suppose no compute, since you need libraries, and libraries have versions, and you can't have them (significantly/major versions) changing under your feet. (Or if they don't have versions they're of course destined to have known security holes.)
(Or it's not even about libraries if you want to say no you don't need libraries - it's just about having to interface with anything.)
I’ve never cordoned off an individual head of cattle and lobotomized it, which is kinda what we do when debugging issues. We take the pod out of rotation, flip a bunch of configs, then give it some traffic to see the new debugging statements.
Before I used Kubernetes for my side projects and only at work I always thought it's hard to operate and very tricky. If you start with an empty "default" cluster and then just add bits when you need them it's actually not that complicated and doesn't feel too heavyweight. I'd suggest to just play around with a simple example and then see how it goes.
There's things that are used in "production" clusters that you don't need at the beginning, like rbac rules, Prometheus annotations etc.
There's now even onprem ECS variants which means not having to pay aws very much and still get the benefit of them running and maintaining the control plane
If you want to learn about containers, distributed workloads, etc, then Nomad is a great option that is easy to learn/adopt piecemeal.
Regarding PaaS stuff like Heroku, the only people I know that are still using that are solo hackers.
It's ridiculous overkill, but I'm looking at a NextCloud server on one of their PiBox hardware servers for the house. (You don't need a PiBox - their stuff will run fine on little instances from AWS/DigitalOcean/Hertzner, etc., or a spare PC you have lying around...)
The API is the main drawcard of k8s in the first place, if you are off in ECS land all you are doing is wasting a bunch of time on a dead-end.
I would instead focus on getting to understand the basics of the API by using a hosted k8s service like GKE or EKS. Stick with some basic manifests, i.e deployments, services, ingress.
Once you have some stuff running you can start learning how it really works and goes together, i.e what are pods, why are pods immutable, why is a replicaset, how does a deployment orchestrate multiple replicasets, what are endpoints, what is the difference between pod readiness/liveliness.
Don't cheat yourself this early in the game, just learn things the right way from the start and save yourself a bunch of work.
Anywhere I’ve worked business always prioritizes high availability and close to zero downtime. No one sees a random delivered feature. But if a node fails at night - everybody knows it. Clients first of all.
We’ve achieved it all almost out of the box with EKS. Setup with Fargate nodes was literally a one-liner of eksctl.
Multiple environments are separated with namespaces. Leader elections between replicas are also easy. Lens is a very simple to use k8s IDE.
If you know what you’re doing with Kubernetes (don’t use EC2 for nodes, they fail randomly), it’s a breeze.
I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our architecture meetings, although he used to be a banking software architect in his previous assignments. Every question I ask ends up in a long 15 minutes monologue by the guy who architected all of it, even the most simple questions. They are soon launching a mobile app (only a mobile app and its corresponding API, not even a website) and they already have more than 60 containers running and talking to each other across three k8s clusters and half of them interact directly with third-parties outside.
Even as I am being paid by the hour, I really feel sad for both the CTO and the developers attending the meeting.
k8s is definitely not for everyone. Google has thousands hardware systems running the same hypervisor, same OS, same container engine and highly specialized stacks of micro-services that need to run by the thousands. And even, I am not sure that k8s would satisfy Google's actual needs tbh.
Ironically, there are some companies that highly benefit from this and they are not necessarily "large" companies. In my case, k8s and devops in general made my life infinitely easier for on-site trainings: those who come with a poorly configured or decade-old laptop can actually enjoy the labs at the same pace than every other attendee.
If I count actually different containers (like, unique PodSpecs, or so), that count drops to ≈30. Even that is "high", and from an architectural standpoint, it isn't really a number I'd use. E.g., we have a simple daemon, but it also has a cronjob associated with it. So it has "2" PodSpecs by that count. But architecturally I'd call it a single thing. How it implements itself, that's up to it.
A lot of our "unique PodSpec" count, too, comes from utility type things that do one thing, and do it well. Logging (which comes from our vendor) is 3 PodSpecs. Metrics is another 3. We have a network latency measurement (literally ping shoved into a container…): PodSpec. A thing that checks certs to ensure they don't expire: PodSpec. HTTP proxy (for SSRF avoidance): PodSpec. A tool that rotates out nodes so that their OSes can be patched: PodSpec. Let's Encrypt automation (a third party too): 3 PodSpecs … but hey, it does its job, and it's a third party tool, so what do I care, so long as it works the API between me and it suffices (and honestly, its logs are pretty good. When it has had problems, I've usually been able to discern why from the logs). DB backup. But most of these don't really add much conceptual overhead; any one is maybe tied (conceptually) to our applications, but not really to all the other utilities. (E.g., there isn't really coupling between, say, the cert renewer and the logging tooling.) A confused/new dev might need to have it explained to them what any of those given tools do, ofc., but … many of them you can just Google.
… in previous jobs where we didn't use Kubernetes, we mostly just ignored a lot of the tasks that these tools handle. E.g., reboot a VM for patches? It was a custom procedure, depending on VM, and what is running on that VM. You needed to understand that, determine what the implications were … etc. And the end result was that reboots just didn't happen. K8s abstracts that (in the form of PDBs, and readiness checks) and can thus automate it. (And ensure that new loads don't need TLC that … an app dev realistically isn't going to be given the time to give.)
If we needed a common thing on every node? That would be rough. We did finally get to having a common base VM image, but even them, all of the per-app VM images would need to be rebased on the newer one, and then all rolled out, and who/how would one even track that? And … in practice, it didn't happen.
The problem I see with this approach is that it has become very difficult to evaluate system-wide topics such as accessibility (or security, or performance) as we constantly deal with a very diverse technological stack and increasingly complex attack surface.
In my opinion, this makes finding competent people who can actually evaluate or assess work almost impossible, unless you hire a Lemming who will run some third-party scanner he found on GitHub: if the scanner doesn't say something is awful or critical, then almost everyone at the table is instantly convinced the system is perfectly robust.
I try to warn my clients by asking them if they think that a judge will be satisfied if they answer "we ran the scanner the other day and the scanner said it was all good" after a customer sues them for failing to comply with a disabilities act.
And with-k8s and without-k8s to me is pretty similar: we vendor or FOSS most of it. The major cloud vendors all have container registries (of … varying quality…); similarly, at a previous company we used S3+a small shim as a Python package store. (We later moved to a vendored solution.)
ELK for logs meant having a daemon set up per VM. Easier in k8s where I can push a DaemonSet to the entire cluster. With VMs … it's a per-app nightmare, really. Even then, that's really not perfect. In practice, in both situations, I feel like you end up having to integrate the apps with the metrics/logs providers. There's just not a common format. Sometimes, there are some libraries, e.g., there's some stuff for Prom's HTTP metrics APIs. Logs … eugh. Nothing amazing; getting structured logging requires per-app changes regardless of what you do. Sure, in either VM or k8s, you can just "suck up syslog/journald / docker logs", but what format are those in? They're not, is the answer, and I find most places do a "one text log per line" assumption (and then have stuff with multiline logs that just gets destroyed/corrupted/lost by the logging daemon) and it misses out on any sort of structured logs. jsonlines through those channels is a slight step up, but usually requires app changes.
It's also a Google engineer caliber solution. Lots of teams said “hey if Google engineers do it then it must be good!”…but forgot that they didn’t have the same in-house talent as Google.
The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing.
for example: https://kubernetes.io/docs/setup/best-practices/cluster-larg...
no more than 5k nodes.
Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that the whole thing is fragile at that size.
Having run a 36k node cluster in 2014, I know that K8s is just not designed for high scale high turnover vaguely complicated job graphs.
I get the allure, but in practice K8s is designed for a specific usecase, and most people don't have that usecase.
for most people you will want either ECS(its good enough, so long as you work around its fucking stupid service scheme) or something similar.
An argument could be made for something like CodeDeploy being better integrated on ECS, but that's more of a "k8s doesn't need CodeDeploy but ECS might" kind of thing. And even then, I wouldn't touch it.
An argument could also be made that upgrading ECS clusters is a bit easier, as the cluster itself, uh, doesn't have a "version". But on Fargate, its pretty painless on EKS, and Fargate ECS tasks do have a "platform version" that generally doesn't have to be worried about (version: LATEST), but is nonzero nonetheless.
Which is really to say that both ECS and EKS puke complexity, because its AWS, but the volume is pretty similar.
[1] https://docs.aws.amazon.com/eks/latest/userguide/aws-load-ba...
K8s is a "middle 80%" scale solution. Its not made to run Google (though Google uses it a ton internally). Its also not made for your average four person startup (though, if you've got that experience internally, its not a bad choice; its not Heroku, but its better than a lot of deployment options out there).
All I'd say is: I've worked in a "scale-up" B2B org under $1M in ARR. We were pretty monolithic; just a backend NodeJS app, and a frontend SSR React app, basic. Five engineers by the time I left. We used K8s (EKS+Fargate). Maybe 50 pods total, across two environments. It was fantastic. We never had to say No to any weird customer, product, or engineering decision which would be difficult in either more managed, or more legacy, systems. Customer wants a custom domain and'll pay $50k for it? Like five lines of YAML and update Route 53, done. Datadog sidecar container so we can ingest some APM traces? Ten lines of YAML copy-pasted from their docs, done. Update the cluster? Click one button in the AWS UI. Every developer wants their own staging environment? Ok, bit more work, but: create some namespaces, retool the CI a bit, we can deploy separate databases in there as well its only staging, actually pretty straightforward.
Half the stuff we did with k8s would have taken three times as long with more native AWS solutions, and some of it probably would have been impossible on something like Heroku. K8s strikes a balance. Its not the simplest thing in the world. I wouldn't grab it on day 1 of a startup's engineering journey. But I wouldn't knock a startup who does.
I can leave the cluster alone for weeks, and it'll take care of itself, my CI will build new docker containers, tooling will start rolling them out across the cluster, if deployments fail they get rolled back, and I get an email, etc.
At some point I was hands-off with the cluster for 6 months and everything kept itself up to date and running just fine.
off the shelf: https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alf...
although that was with something like 6-10k nodes because there was an upper limit to how many dispatches alfred could do because it was single threaded, from the early 90s and not really designed to scale that high
https://renderman.pixar.com/tractor is probably what they use now, or https://www.opencue.io/
but any grid engine style dispatcher/manager will do what you want. It'll give you the primitives to manage wildly larger scale than k8s.
These clusters were on real steel, as elastic clusters were horrendously expensive, and the storage was/is nowhere near fast enough.
Nowadays, I'd use AWS batch, or at a push airflow.
The fundamental design just isn't infinitely scalable, and at a certain point, you might rather have some bulkheads/autonomy or regional diversity.
Folks, listen, if StackOverflow can run on this: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
So can your doctor's appointment website, your little ML app or Notion clone.
"But...". No. You ain't gonna need it. Do some load testing, prove it to yourself. Now, multiply the load by 100x, reserve AWS resources and you're good to go.
The longer answer is: When you switch to Kubernetes, you are introducing _a lot_ of complexity, which, depending on your actual project, might not be inherent complexity. Yes, you get a shiny tool, but you also get a lot of more things to think about and to manage, to run that cluster, which in turn will require, that you get more devops on board.
Sure, there might be projects out there, where Kubernetes is the right solution, but before you switch to it, have a real long hard thinking about that and definitely explore simpler alternatives. It is not like Kubernetes is the only game in town. It is also not like Google invents any wheels with Kubernetes.
Not everyone is Google or Facebook or whatever. We need to stop adopting solutions just because they get hyped and used at big company. We need to look more at our real needs and avoid introducing unnecessary complexity.
Everything has become so abstracted away these days that performance isn’t even a consideration because people don’t understand the full stack. And I don’t mean “full stack” as it’s slung around these days. EG: doing 20 round trips to the database looking up individual records (as each is an object code-side), when you could just do one. Things like that are opaque & many devs wouldn’t even care or know, but a little bit of education can make a huge difference.
I'd wager providing all those things 20 years ago without k8s and CI tools would've required relatively more sysadmins
Yeah, you lost me already. This is a bit like asking why there are other languages besides Java.
When technology makes previous difficult things easy, this technology will be used to do more things that were previously impossible.
I personally haven't seen anyone use k8s to achieve things that were impossible before. They just use it because 1) They think everyone is using it 2) They don't know how to do it any other way
Kubernetes and 'DEVOPS' are the new hotness at non-FANG companies as they are always 5-10 years behind the trends. Expect to see more of it before it goes out of fashion.
Also DevOps is just a title. Nobody read the book, nobody is trying to create what the original guy at Google or whatever had in mind. It is just a all encompassing job doing the same activities that the sysadmin used to do. HR tells companies that they should rename their sysadmin departments to DevOps departments and everything else continues as normal.
-------
Take the development of programming as an analogy:
* Punched cards
* Programming in assembler
* Goto
* Callable procedures
* Proper functions
* Compiled languages (There used to be companies just selling a big C compiler)
* Interpreters/JIT compilation/...
* ...
-------
And here's a similar progression:
* Servers in your basement
* Some server rented where you login via SSH
* Docker/Kubernetes/Clusters in the cloud
* Lambdas and other serverless solutions
* ...
As a sibling comment pointed out: We're still in the stone ages. Somewhere between punch cards and proper functions.
-------
To rephrase it in reversal: "Infrastructure as code has it backwards"
Right now, we manually partition our code, then provision infrastructure and then push our code to this infrastructure.
Instead, we should take a high level language that compiles to the cloud:
Just write your business logic and the the compiler figures out what clusters/services/event-buses/databases/etc to use; It will automatically partition the code, package, build, provision, push, update. And there's even room for something like JIT: Based on the load parts of your logic get, the compiler could switch databases. Also: Automated data migrations based on your code updates. But I guess we'll end up with a big distributed virtual machine that scales infinitely and completely hides the existence of servers.
There's already some glimpses of this future: No-code, the magic pulumi does with lambdas, several language projects that get rid of the file system and just store the AST in a DB, smart contracts where you pay for single computation steps...
-------
But back to the question: Kubernetes/AWS/etc is a lot of work because it's not really THE SOLUTION.
We already have the tools to have a cloud-scale application dynamically redefining and re-allocating its infrastructure. Give those tools some time to mature, and I'm sure the capabilities will be awesome.
A good example is the common database backend. A lot of groups have some RDS instance provisioned someway to somehow serve some frontend. Over time, the number of users grow and the RDS needs more resources provisioned for it. My last 4 jobs, it was always someone's job to check the provisioning or respond to an alert and bump the provisioning up to whatever the next rung was. In the not too distant future, this type of task will be handled by a mildly intelligent CDK application.
I don't see people wanting to give up control to get a platform like that any time soon.