There ARE other ways, but mesos was a freaking mess. I actually liked swarm, but it lost. k8s also brings in pretty novel ideas that destroy proprietary IaaS, such as the nginx (or other) front end ingress routers that I haven't seen implemented meaningfully in a IaaS neutral way.
Sure, you may have a few properties in your k8s yaml that change from AWS vs Google vs Azure vs other k8s hosts. That's not something that takes more than a debugging session to get right. That's HUUUGE progress.
"lost" is too strong of a word. To me Docker swarm mode still holds the upper hand in terms of deployment ease and simplicity.
If anyone just wants to bolt together a couple of nodes without much concern regarding performance then Docker swarm mode is perhaps the best option out there at the moment.
However, this is changing very fast. Ubuntu now ships with microk8s which just works and outperforms Docker swarm mode.
Perhaps Docker swarm's worst enemy is how the project was managed and promoted. They had a tool to trivially setup a cluster of nodes with close to zero challenge, but somehow Kubernetes managed to be pushed as the only option.
And, that covers 99.9% of cases anyway. If you want to deploy across multiple nodes run docker compose on multiple nodes and expose the nodes to each other over a vlan that is managed as part of your actual infrastructure where it has a chance of being actually documented and put into a configuration management system.
> If you want to deploy across multiple nodes run docker compose on multiple nodes and expose the nodes to each other over a vlan that is managed as part of your actual infrastructure
Indeed, and for a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem.
It doesn’t mean Kubernetes isn’t a useful turnkey solution, especially if it’s managed.
Because you press 4 buttons on a cloud console, or write a bit of terraform, and you have a consistent management plane for all of this, one that you can plug into any tool the very large ecosystem.
Then your entire engineering org just deploys things.
Need per-application network security policies? Done.
Per app IAM roles? Done.
Secrets management? Done.
Role based access control? Done.
Isolate services to different pools of machines? Done.
Per-app live-updating configuration files across your entire fleet? Done.
CPU, memory or arbitrary per-app metric based auto scaling? Done.
Want to run all this on cheap spot instances? Done.
Want path based HTTP routing within your cluster? Done.
Want an easy deployment mechanism with versioning and rollbacks? Done.
Want simple and consistent service discovery across all your servers? Done.
Want to get advanced and build custom resources to manage groups of other resources whilst taking advantage of k8s management plane in a consistent manner that just works with all the other tooling? Done.
Want to do all this fairly consistently on any cloud vendor, or even on prem? Done.
Want a large pool of candidates familiar with this stack? Done and done.
All those lead to non trivial savings, so you can actually focus on your product and not building a crappy version of Kubernetes by stringing together docker compose instances.
But yes, if you’ve got a single developer and you’re deploying by ssh-ing into a single machine and running git pull, it’s definitely overkill.
If it really were "100000x" less effort to fix, why are more and more resources being allocated to keep it going.
It's a full time position keeping all the plates spinning in their air, but now It's even harder to debug and reason about. I guess it allows you to quickly start up a new project with some boilerplate configurations though. I wonder if there's a term for that, where you defer things perceived to be "hard" to save time up front. Sort of like a credit card you spend but with technical decisions ...
> why are more and more resources being allocated to keep it going
What does that even mean? Where? Across the entire industry? In specific companies? Empirically how much “more resources”?
Conversely you could ask: if k8s is so terrible and building your own version is so simple, why is the market not doing that?
Everywhere I’ve seen that uses k8s does indeed have full time Devops/platform people, but the ratio to the developers they support is far, far lower than homegrown platforms, as is the time to implement new requirements.
Usually these homegrown solutions get to a specific point in time where there isn’t the internal knowledge to maintain them, and the effort required grows too large. At which point they stagnate. Building on a stagnant, homegrown platform that you don’t know how to improve is not great.
> What's the actual savings if you now just have additional complexity (...)
You don't. You get less complexity, actually. Even with Docker, a Docker stack in Docker swarm mode is that simpler and easier to maintain and to monitor and to update and roll back than an unexplainable and impossible to justify docker-compose up.
With Kubernetes, it can be even easier than that. Just grab a distro like Ubuntu, instal microk8s and deploy a kustomize script right out of the bat.
Are you really running away from complexity or just irrationally hating on a tool just because you're oblivious to what it provides for free?
It may not be shifting the goal posts much because the typical situations that k8s comes up is when you need multiple nodes, so yeah we're IN the 0.1% of k8s users in here.
> foreach serviceX.servers as server { ssh user@ip "cd service-x-project-dir && git pull && ./deploy.sh && service serviceX reload" }
Do you honestly think this sort of thing, hand-waving included, is a better alternative to
$ kubectl apply -k <kustomize script>
I mean, what are your plans to react to one of your serviceX crapping out? How do you know? How do you recover? How do you roll back? Do you expect to throw more of your ad hoc shell scripts to the mix?
Your example is as good as presenting your deployment strategy as
Software is not really random. Mostly, they all fail or none of them fail. But in case only 1 server has a problem. You'd just use a script that removes that server from your load balancer so you can inspect it, fix it, etc..
I dont do rollbacks. If you deploy code that bugs out, you already failed. Because clearly the coders & reviewers did NOT understand the code. Proper testing helps also. And if doom day comes, just deploy a previous commit. It's pretty much the same as what u do with kubernetes. That being said. Kubernetes is indeed much easier for this.
It all comes down to: "do you actually know what you are doing?". If you know what you are doing, all these problems dont happen. If you feel it's important to be able to spin up new servers from scratch every time you deploy. i guess kubernetes is great for that. it's not for me.
As for being able to know what is happening, i do understand that can be impossible sometimes in large companies. Too many developers, too much code. Still, it shouldnt be used as an execuse for breaking the live servers.
Software fails in unforeseen ways, as the goal is for it to not fail but fails anyway. I'm not sure what point you tried to make
> Mostly, they all fail or none of them fail.
No, not really. There is a reason why one of the very basic deployment techniques is to add health checks for each and every service, run a blue-green deployment, and only switch to the new deployment when each and every single health check points to a successful deployment.
The pseudo-code example fails to even do the most basic checks.
I want someone to check me on a strongly held belief, I promise I have an open mind:
There’s nothing wrong with experimenting with k8s in your org. Nothing wrong with k8s, it’s complex yes but that’s not inherently problematic.
But it’s been my experience over the last several jobs that many orgs are throwing mission critical workloads and hinging a lot of people’s productivity onto k8s and asking operators and engineers to ostensibly figure it out as they go. And from rants and vents I see on other tech communities, I’m not the only one making this observation.
Which is sheer madness to me.
Change my mind: unless you have probably, razor sharp engineers who know what they’re doing with k8s enough to give you more than a few days of availability before livened probes start falling off the face of the planet, AND can make this their sole focus, maybe you’re not ready for k8s?
I don't think it's just orgs, as there are a lot of engineers that seem to have bought into the `k8s == awesome` mentality. I like Kubernetes and I think it has a lot to recommend it, but it's not a single silver bullet for all problems.
I would take this a step further to say that an extremely high % (like 95%) of organizations using k8s do not need it and would be much better off not using it.
I think you have a lot of people coming into the general backend/business-logic tech space who are not nearly enough of "computer people" or engineering-minded to actually be responsible for the uptime of a mission critical workload. They might think they are, but they don't know what it actually takes. (Your "operators and engineers", ostensibly, CAN handle this, but they need to be up to the task too.)
K8s among other things is a way of trying to abstract away the most computer/OS/network-literate parts of a software organization, so that you can be productive, as an org, hiring some computer people along with plenty of the readily-available "smart people who can write code." The former keeps your software running, the latter represent your business as code. It's a way of solving for the hiring market you HAVE, rather than the hiring market of 20 yrs ago. But it hinges on those deep-tech people being up to the task.
Yes, I 100% agree with this take. Engineers need to understand how k8s works at least at a surface level to engineer their service topology, and whatever devops/SRE/ops team will support k8s needs to scale up on test workloads and simulated production workloads and understand how to debug the system before it can be operated at scale.
Running k8s at scale is very challenging due to its complexity and comes with a learning curve that you probably don't want to be ramping through in live-site production issues.
It's a big tool for a big problem that most people don't have.
I have seen k8s used in production 3 times: In a very slow, measured, long-term rollout on owned hardware, which was generally speaking quite successful and paid dividends against the previous home-rolled Ansible/Docker based solution with manual container scheduling by improving allocation, moving networking definition into a much more declarative / "shift-left" way where engineers would define their network topology directly, and improving insight into the system using off-the-shelf tools.
I've also seen k8s used in a very basic fashion on GKE in a mostly painless way - basically just send it and it works.
The worst k8s situation I've seen is one where a startup's GKE infrastructure was migrated into a self-hosted k8s cluster which was cobbled together and had never been scaled up before. Nobody understood the failure points of the system, trivial mistakes caused frequent outages, and as engineers lost faith in the system they started blaming k8s for application level issues. Diving headfirst into a complex system with a production workload is a recipe for pain.
> It's a big tool for a big problem that most people don't have.
A thousand times yes. To a lesser extent, I would apply this to containerization generally.
If your needs are met by some Ansible/Chef/Puppet plays/recipes on a VPS, then you should embrace the simplicity and immediate flexibility you have, and just go that route.
I have seen so much relative over engineering to run simple workloads in containers that never needed to be scaled like that in the first place.
I think people want to pretend that deployment best practices "don't matter" with k8s but of course they do
As a start, see how many people just take the biggest docker image they can find, and install several irrelevant packages to run their service. Don't optimize/limit memory or CPU usage. Don't have a service that follows the 12-factor-app not even at the basic level (like fail fast, etc)
> But it’s been my experience over the last several jobs that many orgs are throwing mission critical workloads and hinging a lot of people’s productivity onto <INSERT_TECH_STACK_HERE> and asking operators and engineers to ostensibly figure it out as they go.
Some FANGs even pride themselves on how they get their army of inexperienced SDEs that on average stay 2 to 3 years with the company to JIT-onboard onto anything.
I wouldn't necessarily agree, but curious to hear this thought developed some more, if you'd be willing to indulge me? Maybe I'm missing some implied context here that you have experience with that could change my mind.
Sometimes complexity is unavoidable, but I don't conflate something being complex with a thing being complicated. Complicated systems I would agree, are inherently problematic.
(also, your username just made me very hungry for a falafel sandwich. Excuse me for a moment)
Our ability to build, understand, analyse, evolve, secure and troubleshoot systems depends on our ability to reason about and understand them. The more complex a system is the less we are able to do the aforementioned things.
This is a wonderful point, and it doesn’t mean that some people can’t hold “all of k8s” in their heads. At least theoretically some can, most can’t.
The simpler a system is, typically the easier it is to explain and maintain. In many businesses that takes a front seat. Some do need this level of complexity though. They’re both valid when applied appropriately.
k8s is really just a lot of Linuxisms put together in semi-novel, intelligent, and pluggable way. That list of features u/orf posted is all possible with plain Linux. You can start really minimal k8s, almost like plain containers on Linux. But going from one machine and one container/workload to literally 100k containers, 100s of machines and workloads working together in secure ways is really difficult without the organization that k8s APIs, interfaces, automations, and concepts impose on Linux.
I've helped an 80% new team survive Black Friday and make $10M revenue in 3 hours in a 6 week engagement. I helped a 100% new team build their first CI/CD system to build/run/test their aircraft embedded system in containers. I’ve helped credit card companies scale dozens of clusters to 150k services under management backed by over 250k containers, per cluster.
And it’s the same code that will run on a 10 year old PC or RPi3B as a single instance. It really is a story about the power of Linux.
Kubernetes appears to be continuing to increase in popularity despite a large number of these failure stories. Does anybody know why? What does kubernetes do that docker, AWS, Nomad, et al don't? And, if I'm stuck with kubernetes, is there an accepted "right way" to do kubernetes that will keep me from becoming a cautionary tale?
Principal kubernetes consultant for a large firm here.
Kubernetes uses Docker runtime so that eliminates your first question. AWS also uses Kubernetes, EKS is a modified version of Kubernetes, and most major companies do not use things like Nomad because they lack the ecosystem tooling that something like Kubernetes has.
I think probably the best thing you could do to prevent issues like this is blue-green testing or proper integration testing with your infrastructure repository so you can roll back quickly when you realize that some change you made didn't work correctly. This is all in addition to significant amounts of it monitoring.
> What does kubernetes do that docker, AWS, Nomad, et al don't?
Most of your comparisons are at best apples-to-oranges.
For example, AWS also offers a managed Kubernetes service, which makes your comparison meaningless and just a keyword dump. Worse, AWS also offers its Elastic Container Service.
Docker supports Docker swarm mode, which is easier to setup but inter-node communication is extremely slow and barely supports anything beyond launching containers.
But to answer the question of why bother with Kubernetes, the answer is obvious: Kubernetes is the first coherent abstraction over heterogeneous clusters of COTS hardware, including networking, which just works and offers plenty of very useful features such as reversible deployment history. Deployments are entirely defined in code in a way that you can treat both deployments and infrastructure as cattle.
It seems to me like a vast majority of startups and use cases don't serve a _ton_ of traffic, nor do they do too much heavy data processing. If we're being realistic, most webapps could probably run on a Raspberry Pi and be fine. With this being the case, why is such a heavy-weight and complicated system so popular?
Literally, design by resume. I see it at my current workplace. Top talent guys are fucking around wit K8s even though we really have no use for it and could host the whole shebang on a single EC2 instance and be no worse off for it. But that "won't scale" I'm told. Never mind our startup has a TAM that involves no more than 1,000 customers with maybe a dozen accounts each.
It's not always about traffic scale. Sometimes it's about having a common deployment platform at different locations, on premise. Where relatively simple and small setups still bring a lot of value via pure convenience - suddenly everything is packed into Helm charts, running off basically identical CICD pipelines.
But I guess that's not what "top talent guys" are interested in doing?
Sometimes it's about having a common deployment platform at different locations, on premise.
That’s fair, that’s valid, that doesn’t however strike me as a compelling case to go off and hoist all of your stuff into k8s if there’s other equally performant and fault tolerant methods of running your application that require less operational complexity and cognitive load to triage and resolve failures.
It seems to me the problem you bring up would serve an engineering team well to solve and solidify even if the application isn’t running on k8s
Transferrable experience in both directions, in and out, is a huge reason. Cynics see this as "resume driven development," but the reality is that offering training and an opportunity to use an industry standard tool isn't a bad move for hiring and retention. Additionally, as you grow, you can hire experts in k8s who have "been there before" with scale - at a huge cost, yes, but this is still a value versus something homegrown.
Also, cloud-hosted k8s systems (especially GKE) aren't half bad these days. Starting with one (carefully, to avoid lock-in) and knowing you can expand later is appealing.
Attempting a serious answer, because I agree most things can be run on a Pi and be fine.
Let's say you're a developer who's comfortable making Docker containers and running them locally. One day, you learn that you could go to Azure|Google|$OTHER and spin up something called Kubernetes, push your container, and now you get a trove of features like scaling, availability, health checks, the safety that you can lose a node and stay online. That sounds appealing to me. Of course, the marketing materials are never the same as where the rubber meets the road, but you stand up your service and all is well.
It's an appealing offer. It's why people use Heroku. If all you have to do is one-click a cluster and go, why wouldn't you want to do that?
> It's an appealing offer. It's why people use Heroku. If all you have to do is one-click a cluster and go, why wouldn't you want to do that?
I think you’ve hit upon what seems evident in this discussion to me, but that doesn’t seem to be highlighted well. Even the most ardent Kubernetes advocates here are noting it’s not a panacea. It’s simply solving a need that exists at an immense scale.
Underlying all of that, though, is a need to understand what you’re deploying no matter what it is. Click to deploy is very tempting, but should only be undertaken when you have an understanding of what you’re clicking and the how and why of the deployment process, the technology stack that makes it possible, etc.
59 comments
[ 2.8 ms ] story [ 59.4 ms ] threadYou'd need a k8s aware FTP service that creates load balanced ports on demand.
Which sounds like it could get thorny and complicated compared to a pair of AWS EC2 instances over an S3 filesystem (Or use AWS's hosted FTP service?)
How do you expect to deploy services in a cluster of multiple nodes communicating over a private software-defined network, then?
Sure, you may have a few properties in your k8s yaml that change from AWS vs Google vs Azure vs other k8s hosts. That's not something that takes more than a debugging session to get right. That's HUUUGE progress.
"lost" is too strong of a word. To me Docker swarm mode still holds the upper hand in terms of deployment ease and simplicity.
If anyone just wants to bolt together a couple of nodes without much concern regarding performance then Docker swarm mode is perhaps the best option out there at the moment.
However, this is changing very fast. Ubuntu now ships with microk8s which just works and outperforms Docker swarm mode.
Perhaps Docker swarm's worst enemy is how the project was managed and promoted. They had a tool to trivially setup a cluster of nodes with close to zero challenge, but somehow Kubernetes managed to be pushed as the only option.
And, that covers 99.9% of cases anyway. If you want to deploy across multiple nodes run docker compose on multiple nodes and expose the nodes to each other over a vlan that is managed as part of your actual infrastructure where it has a chance of being actually documented and put into a configuration management system.
Indeed, and for a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem.
It doesn’t mean Kubernetes isn’t a useful turnkey solution, especially if it’s managed.
Then your entire engineering org just deploys things.
Need per-application network security policies? Done.
Per app IAM roles? Done.
Secrets management? Done.
Role based access control? Done.
Isolate services to different pools of machines? Done.
Per-app live-updating configuration files across your entire fleet? Done.
CPU, memory or arbitrary per-app metric based auto scaling? Done.
Want to run all this on cheap spot instances? Done.
Want path based HTTP routing within your cluster? Done.
Want an easy deployment mechanism with versioning and rollbacks? Done.
Want simple and consistent service discovery across all your servers? Done.
Want to get advanced and build custom resources to manage groups of other resources whilst taking advantage of k8s management plane in a consistent manner that just works with all the other tooling? Done.
Want to do all this fairly consistently on any cloud vendor, or even on prem? Done.
Want a large pool of candidates familiar with this stack? Done and done.
All those lead to non trivial savings, so you can actually focus on your product and not building a crappy version of Kubernetes by stringing together docker compose instances.
But yes, if you’ve got a single developer and you’re deploying by ssh-ing into a single machine and running git pull, it’s definitely overkill.
Except this breaks less than a homegrown solution would with 100,000x less effort fix, build or maintain.
It's a full time position keeping all the plates spinning in their air, but now It's even harder to debug and reason about. I guess it allows you to quickly start up a new project with some boilerplate configurations though. I wonder if there's a term for that, where you defer things perceived to be "hard" to save time up front. Sort of like a credit card you spend but with technical decisions ...
What does that even mean? Where? Across the entire industry? In specific companies? Empirically how much “more resources”?
Conversely you could ask: if k8s is so terrible and building your own version is so simple, why is the market not doing that?
Everywhere I’ve seen that uses k8s does indeed have full time Devops/platform people, but the ratio to the developers they support is far, far lower than homegrown platforms, as is the time to implement new requirements.
Usually these homegrown solutions get to a specific point in time where there isn’t the internal knowledge to maintain them, and the effort required grows too large. At which point they stagnate. Building on a stagnant, homegrown platform that you don’t know how to improve is not great.
You don't. You get less complexity, actually. Even with Docker, a Docker stack in Docker swarm mode is that simpler and easier to maintain and to monitor and to update and roll back than an unexplainable and impossible to justify docker-compose up.
With Kubernetes, it can be even easier than that. Just grab a distro like Ubuntu, instal microk8s and deploy a kustomize script right out of the bat.
Are you really running away from complexity or just irrationally hating on a tool just because you're oblivious to what it provides for free?
there u go. As for SDN, why do u even need that? Some simple scripts that manage your server count and their configurations over ssh is all u need.
Do you honestly think this sort of thing, hand-waving included, is a better alternative to
$ kubectl apply -k <kustomize script>
I mean, what are your plans to react to one of your serviceX crapping out? How do you know? How do you recover? How do you roll back? Do you expect to throw more of your ad hoc shell scripts to the mix?
Your example is as good as presenting your deployment strategy as
$ load ""
I dont do rollbacks. If you deploy code that bugs out, you already failed. Because clearly the coders & reviewers did NOT understand the code. Proper testing helps also. And if doom day comes, just deploy a previous commit. It's pretty much the same as what u do with kubernetes. That being said. Kubernetes is indeed much easier for this.
It all comes down to: "do you actually know what you are doing?". If you know what you are doing, all these problems dont happen. If you feel it's important to be able to spin up new servers from scratch every time you deploy. i guess kubernetes is great for that. it's not for me.
As for being able to know what is happening, i do understand that can be impossible sometimes in large companies. Too many developers, too much code. Still, it shouldnt be used as an execuse for breaking the live servers.
Software fails in unforeseen ways, as the goal is for it to not fail but fails anyway. I'm not sure what point you tried to make
> Mostly, they all fail or none of them fail.
No, not really. There is a reason why one of the very basic deployment techniques is to add health checks for each and every service, run a blue-green deployment, and only switch to the new deployment when each and every single health check points to a successful deployment.
The pseudo-code example fails to even do the most basic checks.
Hell, even Ansible supports those.
There’s nothing wrong with experimenting with k8s in your org. Nothing wrong with k8s, it’s complex yes but that’s not inherently problematic.
But it’s been my experience over the last several jobs that many orgs are throwing mission critical workloads and hinging a lot of people’s productivity onto k8s and asking operators and engineers to ostensibly figure it out as they go. And from rants and vents I see on other tech communities, I’m not the only one making this observation.
Which is sheer madness to me.
Change my mind: unless you have probably, razor sharp engineers who know what they’re doing with k8s enough to give you more than a few days of availability before livened probes start falling off the face of the planet, AND can make this their sole focus, maybe you’re not ready for k8s?
K8s among other things is a way of trying to abstract away the most computer/OS/network-literate parts of a software organization, so that you can be productive, as an org, hiring some computer people along with plenty of the readily-available "smart people who can write code." The former keeps your software running, the latter represent your business as code. It's a way of solving for the hiring market you HAVE, rather than the hiring market of 20 yrs ago. But it hinges on those deep-tech people being up to the task.
Running k8s at scale is very challenging due to its complexity and comes with a learning curve that you probably don't want to be ramping through in live-site production issues.
It's a big tool for a big problem that most people don't have.
I have seen k8s used in production 3 times: In a very slow, measured, long-term rollout on owned hardware, which was generally speaking quite successful and paid dividends against the previous home-rolled Ansible/Docker based solution with manual container scheduling by improving allocation, moving networking definition into a much more declarative / "shift-left" way where engineers would define their network topology directly, and improving insight into the system using off-the-shelf tools.
I've also seen k8s used in a very basic fashion on GKE in a mostly painless way - basically just send it and it works.
The worst k8s situation I've seen is one where a startup's GKE infrastructure was migrated into a self-hosted k8s cluster which was cobbled together and had never been scaled up before. Nobody understood the failure points of the system, trivial mistakes caused frequent outages, and as engineers lost faith in the system they started blaming k8s for application level issues. Diving headfirst into a complex system with a production workload is a recipe for pain.
A thousand times yes. To a lesser extent, I would apply this to containerization generally.
If your needs are met by some Ansible/Chef/Puppet plays/recipes on a VPS, then you should embrace the simplicity and immediate flexibility you have, and just go that route.
I have seen so much relative over engineering to run simple workloads in containers that never needed to be scaled like that in the first place.
As a start, see how many people just take the biggest docker image they can find, and install several irrelevant packages to run their service. Don't optimize/limit memory or CPU usage. Don't have a service that follows the 12-factor-app not even at the basic level (like fail fast, etc)
Some FANGs even pride themselves on how they get their army of inexperienced SDEs that on average stay 2 to 3 years with the company to JIT-onboard onto anything.
I wouldn't necessarily agree, but curious to hear this thought developed some more, if you'd be willing to indulge me? Maybe I'm missing some implied context here that you have experience with that could change my mind.
where does the opposite idea stem from?
Good question. For me personally, it stems from my subscription to the dichotomy of accidental vs. essential complexity as written about here (and other places) https://simplicable.com/new/accidental-complexity-vs-essenti...
Sometimes complexity is unavoidable, but I don't conflate something being complex with a thing being complicated. Complicated systems I would agree, are inherently problematic.
(also, your username just made me very hungry for a falafel sandwich. Excuse me for a moment)
The simpler a system is, typically the easier it is to explain and maintain. In many businesses that takes a front seat. Some do need this level of complexity though. They’re both valid when applied appropriately.
I've helped an 80% new team survive Black Friday and make $10M revenue in 3 hours in a 6 week engagement. I helped a 100% new team build their first CI/CD system to build/run/test their aircraft embedded system in containers. I’ve helped credit card companies scale dozens of clusters to 150k services under management backed by over 250k containers, per cluster.
And it’s the same code that will run on a 10 year old PC or RPi3B as a single instance. It really is a story about the power of Linux.
https://news.ycombinator.com/item?id=20163500
https://news.ycombinator.com/item?id=18953647
https://news.ycombinator.com/item?id=26106080
Kubernetes uses Docker runtime so that eliminates your first question. AWS also uses Kubernetes, EKS is a modified version of Kubernetes, and most major companies do not use things like Nomad because they lack the ecosystem tooling that something like Kubernetes has.
I think probably the best thing you could do to prevent issues like this is blue-green testing or proper integration testing with your infrastructure repository so you can roll back quickly when you realize that some change you made didn't work correctly. This is all in addition to significant amounts of it monitoring.
Most of your comparisons are at best apples-to-oranges.
For example, AWS also offers a managed Kubernetes service, which makes your comparison meaningless and just a keyword dump. Worse, AWS also offers its Elastic Container Service.
Docker supports Docker swarm mode, which is easier to setup but inter-node communication is extremely slow and barely supports anything beyond launching containers.
But to answer the question of why bother with Kubernetes, the answer is obvious: Kubernetes is the first coherent abstraction over heterogeneous clusters of COTS hardware, including networking, which just works and offers plenty of very useful features such as reversible deployment history. Deployments are entirely defined in code in a way that you can treat both deployments and infrastructure as cattle.
What does this analogy mean?
But I guess that's not what "top talent guys" are interested in doing?
That’s fair, that’s valid, that doesn’t however strike me as a compelling case to go off and hoist all of your stuff into k8s if there’s other equally performant and fault tolerant methods of running your application that require less operational complexity and cognitive load to triage and resolve failures.
It seems to me the problem you bring up would serve an engineering team well to solve and solidify even if the application isn’t running on k8s
Also, cloud-hosted k8s systems (especially GKE) aren't half bad these days. Starting with one (carefully, to avoid lock-in) and knowing you can expand later is appealing.
Let's say you're a developer who's comfortable making Docker containers and running them locally. One day, you learn that you could go to Azure|Google|$OTHER and spin up something called Kubernetes, push your container, and now you get a trove of features like scaling, availability, health checks, the safety that you can lose a node and stay online. That sounds appealing to me. Of course, the marketing materials are never the same as where the rubber meets the road, but you stand up your service and all is well.
It's an appealing offer. It's why people use Heroku. If all you have to do is one-click a cluster and go, why wouldn't you want to do that?
I think you’ve hit upon what seems evident in this discussion to me, but that doesn’t seem to be highlighted well. Even the most ardent Kubernetes advocates here are noting it’s not a panacea. It’s simply solving a need that exists at an immense scale.
Underlying all of that, though, is a need to understand what you’re deploying no matter what it is. Click to deploy is very tempting, but should only be undertaken when you have an understanding of what you’re clicking and the how and why of the deployment process, the technology stack that makes it possible, etc.