My current company is split... maybe 75/25 (at this point) between Kubernetes and a bespoke, Ansible-driven deployment system that manually runs Docker containers on nodes in an AWS ASG and will take care of deregistering/reregistering the nodes with the ALB while the containers on a given node are getting futzed with. The Ansible method works remarkably well for it's age, but the big thing I use to convince teams to move to Kubernetes is that we can take your peak deploy times from, say, a couple hours down to a few minutes, and you can autoscale far faster and more efficiently than you can with CPU-based scaling on an ASG.
From service teams that have done the migrations, the things I hear consistently though are:
- when a Helm deploy fails, finding the reason why is a PITA (we run with --atomic so it'll rollback on a failed deploy. What failed? Was it bad code causing a pod to crash loop? Failed k8s resource create? who knows! have fun finding out!)
- they have to learn a whole new way of operating, particularly around in-the-moment scaling. A team today can go into the AWS Console at 4am during an incident and change the ASG scaling targets, but to do that with a service running in Kubernetes means making sure they have kubectl (and it's deps, for us that's aws-cli) installed and configured, AND remembering the `kubectl scale deployment X --replicas X` syntax.
I haven’t used kubernetes in a few years, but do they have a good UI for operations? Your example of the AWS console where you can just log in and scale something in the UI but for kubernetes. We run something similar on AWS right now, during an incident we log into the account with admin access to modify something and then go back to configure that in the CDK post incident.
AWS has a UI for resources in the cluster but it relies on the IAM role you're using in the console to have configured perms in the cluster, and our AWS SSO setup prevents that from working properly (this isn't usually the case for AWS SSO users, it's a known quirk of our particular auth setup between EKS and IAM -- we'll fix it sometime).
I have to say that when you have more buy in from delivery teams and adoption of HPAs your system can become more harmonious overall. Each team can monitor and tweak their services, and many services are usually connected upstream or downstream. When more components can ebb and flow according to the compute context then the system overall ebbs and flows better. #my2cents
Personally, I don't like Helm. I think for the vast majority of usecases where all you need is some simple templating/substitution, it just introduces way more complexity and abstraction than it is worth.
I've been really happy with just using `envsubst` and environment variables to generate a manifest at deploy time. It's easy with most CI systems to "archive" the manifest, and it can then be easily read by a human or downloaded/applied manually for debugging with. Deploys are also just `cat k8s/${ENV}/deploy.yaml | envsubt > output.yaml && kubectl apply -f output.yaml`
I've also experimented with using terraform. It's actually been a good enough experience that I may go fully with terraform on a new project and see how it goes.
You might like kubernetes kustomize if you don't care for helm (IMO, just embrace helm, you can keep your charts very simple and it's straight forward). Kustomize takes a little getting used to, but it's a nice abstraction and widely used.
I cannot recommend terraform. I use it daily, and daily I wish I did not. I think Pulumi is the future. Not as battle tested, but terraform is a mountain of bugs anyway, so it can't possibly be worse.
Just one example where terraform sucks: You cannot both deploy a kubernetes cluster (say an EKS/AKS cluster) and then use kubernetes_manifest provider in a single workspace. You must do this across two separate terraform runs.
The problem with bespoke, homegrown, and DIY isn't that the solutions are bad. Often, they are quite good—excellent, even, within their particular contexts and constraints. And because they're tailored and limited to your context, they can even be quite a bit simpler.
The problem is that they're custom and homegrown. Your organization alone invests in them, trains new staff in them, is responsible for debugging and fixing when they break, has to re-invest when they no longer do all the things you want. DIY frameworks ultimately end up as byzantine and labyrinthine as Kubernetes itself. The virtue of industry platforms like Kubernetes is, however complex and only half-baked they start, over time the entire industry trains on them, invests in them, refines and improves them. They benefit from a long-term economic virtuous cycle that DIY rarely if ever can. Even the longest, strongest, best-funded holdouts for bespoke languages, OSs, and frameworks—aerospace, finance, miltech—have largely come 'round to COTS first and foremost.
I understand where most of the complexity in K8S comes from, but it still horrifies and offends me and I hate it. But I don't think it's Kubernetes' fault directly. I think the problem is deeper in the foundation. It comes from the fact that we are trying to build modern, distributed, high availability, incrementally upgradeable, self-regulating systems on a foundation of brittle clunky 1970s operating systems that are not designed for any of that.
The whole thing is a bolt-on that has to spend a ton of time working around the limitations of the foundation, and it shows.
Unfortunately there seems to be zero interest in fixing that and so much sunk cost in existing Unix/Posix designs that it seems like we are completely stuck with a basic foundation of outdated brittleness.
What I think we need:
* An OS that runs hardware-independent code (WASM?) natively and permits things like hot updates, state saving and restoration, etc. Abstract away the hardware.
* Native built-in support for clustering, hot backups, live process migration between nodes, and generally treating hardware as a pure commodity in a RAIN (redundant array of inexpensive nodes) configuration.
* A modern I/O API. Posix I/O APIs are awful. They could be supported for backward compatibility via a compatibility library.
* Native built-in support for distributed clustered storage with high availability. Basically a low or zero config equivalent of Ceph or similar built into the OS as a first class citizen.
* Immutable OS that installs almost instantly on hardware, can be provisioned entirely with code, and where apps/services can be added and removed with no "OS rot." The concept of installing software "on" the OS needs to be killed with fire.
* Shared distributed network stack where multiple machines can have the same virtual network interfaces, IPs, and open TCP connections can migrate. Built-in load balancing.
I'm sure people around here can think of more ideas that belong in this list. These are not fringe things that are impossible to build.
Basically you should have an immutable image OS that turns many boxes into one box and you don't have to think about it. Storage is automatically clustered. Processes automatically restart or, if a hardware fault is detected in time, automatically migrate.
There were efforts to build such things (Mosix, Plan 9, etc.) but they were bulldozed by the viral spread of free Unix-like OSes that were "good enough."
Edit:
That being said, I'm not saying Kubernetes is good software either. The core engine is actually decent and as the OP said has a lot of complexity that's needed to support what it does. The ugly nasty disgusting parts are the config interface, clunky shit like YAML, and how generally arcane and unapproachable and ugly the thing is to actually use.
I just loathe software like this. I feel the same way about Postgres and Systemd. "Algorithmically" they are fine, but the interface and the way you use them is arcane and makes me feel like I'm using a 70s mainframe on a green VT220 monitor.
Either these things are designed by the sorts of "hackers" who like complexity and arcane-ness, or they're hacks that went viral and matured into global infrastructure without planning. I think it's a mix of both... though in the case of Postgres it's also that the project is legitimately old. It feels like old-school Unix clunkware because it is.
I'm not entirely convinced that there isn't a better way. With AWS Lambda and alternatives able to run containers on demand, and OpenFaas, they all point to "a better way".
[Edit] Parent comment is almost entirety completely different after that edit to what I responded to. But I think my point still stands.
One day, hopefully in my lifetime, we shall see it.
Yeah I do think lambda-style coding where you move away from the idea of processes toward functions and data are another possibly superior way.
The problem is that right now this gets you lock-in to a proprietary cloud. There are some loose standards but the devil's in the details and once you are deployed somewhere it's damn hard to impossible to move without serious downtime and fixing.
I don't know it either, but a vague understanding I got in the past was the language itself wasn't very user-friendly. I think Elixir was supposed to solve that.
Completely agree, but that's where OpenFaas (or another open standard) comes in.
Hopefully we should get OpenFaas and Lambda, in the same way we have ECS and EKS. Standardised ways to complete tasks, rather than managing imaginary servers.
Agreed. If Linux were a distributed OS, people would just be running a distro with systemd instead of K8s. (Of course, systemd is just another kubernetes, but without the emphasis on running distributed systems)
That whole concept is bizarre. It's like wanting to fly, so rather than buy a plane, you take a Caprice Classic and try to make it fly.
If CoreOS actually wanted to make distributed computing easier, they'd make patches for the Linux kernel (or make an entirely different kernel). See the many distributed OS kernels that were made over 20 years ago. But that's a lot of work. So instead they tried to go the cheap and easy route. But the cheap and easy route ends up being much shittier.
There's no commercial advantage to building a distributed OS, which is why no distributed OS is successful today. You would need a crazy person to work for 10 years on a pet project until it's feature-complete, and then all of a sudden everyone would want to use it. But until it's complete, nobody would use it, and nobody would spend time developing it. Even once it's created, if it's not popular, still nobody will use it (you can use Plan9 today, but nobody does).
I’ll add to this:
Boot and compute need to be entirely disconnected from one another.
If I have a block of storage that boots on one system, it should boot on another. An OS should poll for available services, allow them to define APIs and be utilized, but no bloat should be added to a base to support elective services.
Everything should be microkernel too, but now I’m just venting.
This matches our experience as well. As long as you treat your managed k8s cluster as autoscaling-group as-a-service you'll do fine.
k8s's worst property is that it's a cleverness trap. You can do anything in k8s whether it's sane to do so or not. The biggest guardrail against falling into is managing your k8s with terraform-ish so that you don't find yourself in a spot where "effort to do it right" >> "effort to hack in YAML" and finding your k8s cluster becoming spaghetti.
Re: cleverness trap. I feel like this is the tragedy of software development. We like to be seen as clever. We are doing "hard" things. I have way more respect for engineers that do "simple" things that just work using boring tech and factor in whole lifecycle of the product.
Not everyone has money to burn, even back in ZIRP era.
And before you trot out wages for experienced operations team - I've regularly dealt with it being cheaper to pay for one or two very experienced people than deal with AWS bill.
For the very simple reason that cloud provider's prices are scaled to US market and not everyone has US money levels.
Sorry, I could have explained that better. The biggest value add that k8s has is that it gives you as many or as few autoscaling groups as you need at a given time using only a single pool (or at least fewer pools) of heterogeneous servers. There's lots of fine print here but it really does let you run the same workloads on less hardware and to me that's the first and last reason you should be using it.
I wouldn't start with k8s and instead opt for ASGs until you reach the point where you look at your AWS account and see a bunch of EC2 instances sitting underutilized.
> Above I alluded to the fact that we briefly ran ephemeral, interactive, session-lived processes on Kubernetes. We quickly realized that Kubernetes is designed for robustness and modularity over container start times.
Is there a clear example of this? E.g. is kubernetes inherently unable to start a pod (assuming the same sequence of events, e.g. warm/cold image with streaming enabled) under 500ms, 1s etc?
I am asking this as someone who spent quite a bit of time and wasn't able to bring it down 2s< mark, which eventually led us to rewrite the latency sensitive parts to use Nomad. But we are currently in a state where we are re-considering kubernetes for its auxilary tooling benefits and would love to learn more if anyone had experiences with starting and stopping thousands of pods with the lowest possible latencies without caring for utilization or placement but just observable boot latencies.
I do believe that with the right knowledge of Kubernetes internals it's probably possible to get k8s cold start times competitive with where we landed without Kubernetes (generally subsecond, often under 0.5s depending on how much the container does before passing a health check), but we'd have to understand k8s internals really well and would have ended up throwing out much of what already existed. And we'd probably end up breaking most of the reasons for using Kubernetes in the first place in the process.
Not much internals needed, but actual in depth understanding of Pod kube-api plus at least basics of how scheduler, kubelet, and kubelet drivers interact.
Big possible win is custom scheduling, but barely anyone seems to know it exists
Yeah, looking into writing a scheduler was basically where we stepped back and said “if we write this ourselves, why not the rest, too”. As I see it, the biggest gains that we were able to get were by making things happen in parallel that would by default happen in sequence, and optimizing for the happy path instead of optimizing for reducing failure. In Kubernetes it's reasonable to have to wait for a dozen things to serially go through RAFT consensus in etcd before the pod runs, but we don't want that.
(I made up the dozen number, but my point is that that design would be perfectly acceptable given Kubernetes' design constraints)
Not surprising to me. People are complaining about how difficult it is to know k8s when you talk about the basic default objects. Getting into the weeds of how the api and control plane work (especially since it has little impact on day to day dev) is something devs tend to just avoid.
Honestly, devs of the applications that run on top probably should not have to worry about it. Instead have a platform team provide the necessary features.
Yeah, with plain Kubernetes I'd also see the practical limit around ~0.5s. If you are on GKE Autopilot where you also have little control over node startup there is likely also a lot more unpredictability.
Something like Knative can allow for faster startup times if you follow the common best-practices (pre-fetching images, etc.), but I'm not sure if it supports enough of the session-related feature that you were probably looking for to be a stand-in for Plane.
I see Kubernetes the same way as git. Elegant fundamental design, but the interface to it is awful.
Kubernetes is designed to solve big problems and if you don't have those problems, you're introducing a tonne of complexity for very little benefit. An ideal orchestrator would be more composable and not introduce more complexity than needed for the scale you're running at. I'd really like to see a modern alternative to K8S that learns from some of its mistakes.
I was once talking to an ex google site reliability engineer. He said there are maybe a handful of companies in the world that _need_ k8s. I tend to agree. A lot of people practice hype driven development.
You either recreate a less reliable version of kubernetes for workload ops or you go all in on your cloud provider and hope they'll be responsible for your destiny.
Vanilla Kubernetes is just enough abstraction to avoid both of those situations.
Doesn't really mesh with my experience, especially the longer k8s been out.
It can be cheaper to depend on cloud provider to ship some features, but with tools like crossplane you can abstract that out so developers can just "order" a database service etc. for their application.
At this point I found out that k8s knowledge is more portable, whereas your trove of $VENDOR_1 knowledge might suddenly have issues because for reasons outside of your capacity to control there's now big spending contract signed with $VENDOR_2 and a mandate to move.
And with smaller companies I tend to find k8s way more cost effective. I pulled things I wouldn't be able to fit in a budget otherwise.
I joined a team that used AWS without kubernetes. Thousands of fragile weird python and bash scripts. Deployment was always such a headache.
A few months later I transitioned the team to use containers with proper CI/CD and EKS with Terraform and Argo CD. The team and also the managers like it, since we could deploy quite quickly.
If you're on AWS, yeah, I'd say just use ECS until you need more complexity. Our ECS deployments have been unproblematic for years now.
Our K8s clusters never goes more than a couple days without some sort of strange issue popping up. Arguably it could be because my company outsourced maintenance of it to an army of idiots. But K8s is a tool that is only as good as the operator, and competence can be hard to come by at some companies.
Agreed. But if you're already on AWS, I'd say the quality floor is already higher than the potential at 95%+ of other companies.
So I say unless you're at a company that pays top salaries for the top 5% of engineering talent, you're probably better off just using the AWS provided service.
I used to have a saying back when Heroku was more favourable, is that you use Heroku because you want to go bankrupt. AWS is at times similar.
Depending on your local market, AWS bills might be way worse than the cost of few bright ops people who will let you choose from offerings including running dev envs on random assortment of dedicated servers and local e-waste escapees
Cloud run, etc, but there seem to be some biggish gaps in what those tools can do (probably because if deploying a container was too easy the cloud providers would lose loads of profit).
I honestly think docker compose is the best default option for single-machine orchestration. The catch is that you either need to do some scripting to get fully automated zero downtime deploys. I have to imagine someone will eventually figure out a way to trivialize that, if they haven't already. Or, you could just do the poor man's zero downtime deploy: run two containers, deploy container a, wait for it to be ready, then deploy container b, and let the reverse proxy do the rest.
Docker Swarm takes the Compose format and takes it to multi-node clusters with load balancing, while keeping things pretty simple and manageable, especially with something like Porainer!
For larger scale orchestratiom, Hashicorp Nomad can also be a notable contender, while in some ways still being simpler than Kubernetes.
And even when it comes to Kubernetes, distros like K3s and tools like Portainer or Rancher can keep managing the cluster easy.
If you want to stick on one machine, you can always just use a single node Docker Swarm to get the fully automated zero downtime deploys you want with Docker Compose:
And that hype is in large part created by Google and other cloud vendors.
To be honest I hardly see any reasonable/actionable advice from Cloud/SAAS vendors. Either it is to sell their stuff or generic stuff like "One should be securing / monitoring their stuff running in prod". Oh wow, never thought or done any such thing before.
That might be true, but unfortunately the state of the art infrastructure tooling is mostly centered around k8s. This means that companies choose k8s (or related technologies like k3s, Microk8s, etc.) not because they strictly _need_ k8s, but because it improves their workflows. Otherwise they would need to invest a disproportionate amount of time and effort adopting and maintaining alternative tooling, while getting an inferior experience.
Choosing k8s is not just based on scaling requirements anymore. There are also benefits of being compatible with a rich ecosystem of software.
Continuous deployment systems like ArgoCD and Flux, user friendly local development environments with tools like Tilt, novel networking, distributed storage, distributed tracing, etc. systems that are basically plug-and-play, etc. Search for "awesome k8s" and you'll get many lists of these.
It's surely possible to cobble all of this together without k8s, but k8s' main advantage is exposing a standardized API that simplifies managing this entire ecosystem. It often makes it worth the additional overhead of adopting, understanding and managing k8s itself.
I push for k8s because I know it. Why not use something that I know how to use? I know how to quickly set up a cluster, what to deploy, and teach other team members about fundamentals.
How many people out there really need C# or object oriented programming?
The argument you present might be valid if you decide to use a tech stack prior having much experience with it.
If you're expecting app/FE devs to have to learn it you're putting a ton of barriers in their way in terms of deploying. Just chucking a container on a non-k8s managed platform (e.g. Cloud Run) would be much simpler, and no pile of bash scripts.
PaaSes are for companies with money to burn, most of the time. A good k8s team (even a single person, to be quite honest) is going to work towards providing your application teams with simple templates to let them deploy their software easily. Just let them do it.
Also, in my experience, you either have to spend ridiculous amounts of money on SaaS/PaaS, or you find that you have to host a lot more than just your application and suddenly the deployment story becomes more complex.
Depending on where you are and how much you're willing to burn money, you might find out that k8s experts are cheaper than the money saved by not going PaaS.
> If you're expecting app/FE devs to have to learn it
Why would anyone expect it? It's not their job, is it? We don't expect backend devs to know frontend and vice-versa, or any of them to have AWS certification. Why would it be different with k8s?
> Just chucking a container on a non-k8s managed platform (e.g. Cloud Run) would be much simpler, and no pile of bash scripts.
Simpler to deploy, sure, but not to actually run it seriously in the long term. Though, if we are talking about A container (as in singular), k8s would indeed be some serious over-engineering
Just a thought as well in my corpo experience: Unfortunately, there are some spaces that distribute solutions as k8s-only... Which sucks. I've noticed this mostly in the data science/engineering world. These are solutions that could be easily served up in a small docker compose env. The complexity/upsell/devops BS is strong.
To add insult to injury, I've seen more than one use IaC cloud tooling as an install script vs a maintainable and idempotent solution. It's all quite sad really.
It's a dumb statement especially from an SRE, it's typically a comment from people that don't understand k8s and think that k8s is only there to have the SLA of Google.
For most use case k8s is not there to give you HA but to give you a standard way of deploying a stack, that being on the cloud or on prem.
He understood it fully, he was running a multi day course on it when I spoke to him. He was candid about the tech, most of us where there at the behest of our orgs.
In my personal experience, Google SREs as well as k8s devs sometimes didn't grok how wide k8s usability was - they also can be blind to financial aspects of companies living outside of Silly Valley.
Most companies in the world don't need to develop software. Software development itself is hype. But there's lots of money in it, despite no actual value being created most of the time.
Kubernetes scales down pretty well. I don't use network layers or crazy ingress setups. I keep it simple and Kubernetes works great.
What's wonderful is that when I work on multiple clouds, my knowledge transfers just fine. I don't think of the AWS solution or the GCS solution, I use the same kubectl to check out both, view logs, inspect and fix.
Even when I got tired of waiting for GKE to spin up a node, running Github actions on a self-hosted microk8s meant instant pod starts and very little fuss. But using Kubernetes meant I got to take advantage of the Github operator, which let me reuse the same machine for multiple builds without the headaches.
When I want to run some open source, I often find a helm chart the helps me get set up. Nowadays running open source packages can involve all kinds of dependencies, but getting it running on a k8s cluster to check it out, or even in prod, is a relatively straight forward editing of some values files. I've recently ran Uptrace and Superset that way. They're not a bajillion requests per second setups, they don't have to be, and it was far easier to set up than most methods.
I would say your friend is right. Few people _need_ k8s but it's one interface to a bunch of complicated proprietary stuff. I can know core small, core set of k8s tools really well and forget half of the junk that I ever knew about public clouds. It's all the same patterns, transferable and reliable.
> It’s also worth noting that we don’t administer Kubernetes ourselves
This is the key point. Even getting to the point where I could install Kubernetes myself on my own hardware took weeks, just understanding what hardware was needed and which of the (far too many) different installers I had to use.
OT: Can something be done about HN commenting culture so that the comments stay more on topic?
Some technologies (like Kubernetes) tend to attract discussions where half of the commenters completely ignore the original article, so we end up having a weekly thread about Kubernetes where the points of the article (which are interesting) are not able to be discussed because they are drowned out by the same unstructured OT discussions?
At the time of this posting there are ~20 comments with ~2 actually having anything to do with the points of the article rather than Kubernetes in general.
Having read the article, isn't the point of the article kubernetes in general and what the author prescribes you sign up for/avoid?
Discussions of k8s pitfalls and successes in general seems to be very much in line with what the article is advocating. And, to that point, there's frankly just not a whole lot interesting in this article for discussion "We avoid yaml and operators"... Neat.
> Having read the article, isn't the point of the article kubernetes in general and what the author prescribes you sign up for/avoid?
Yeah, and I think that provides a good basis to discussion, where people can critique/discuss whether the evaluation that the author has made are correct (which a few comments are doing). At the same time a lot of that discussion is being displaced by what I would roughly characterize as "general technology flaming" which isn't going anywhere productive.
What you're seeing is the early-crowd. With most (not all) posts, comments will eventually rise to the top that are more what you're looking for. IME it usually takes a couple hours. If it's a post where I really want to read the relevant comments, I'll usually come back at least 8 to 12 hours later and there's usually some good ones to choose from. Even topics like Apple that attract the extreme lovers and haters tend to trend this direction
The solution to that is to flag boring/generic articles and/or post/upvote more specific, interesting articles. Generic articles produce generic, mostly repetitive comments but then again that's what the material the commenters are given.
Yeah, I disagree with the OP on the dangers there. They work fairly well for us and aren't the source of headache. Though, I still try and teach my dev teams that "just because bitnami puts in variables everywhere, doesn't mean you need to. We aren't trying to make these apps deployable on homelabs."
> But we often do multiple deploys per day, and when our products break, our customer’s products break for their users. Even a minute of downtime is noticed by someone.
Kubernetes might be the right tool for the job if we accept that this is a necessary evil. But maybe it's not? The idea that I might fail to collaborate with you because a third party failed because a fourth party failed kind of smells like a recipe for software that breaks all the time.
It really comes down to, I don't ever want to have the conversation “is this a good time to deploy, or should we wait until tonight when there’s less usage”. We have had some periods where our system was more fragile, and planning our days around the least-bad deployment window was a time suck, and didn't scale to our current reality of round-the-clock usage.
You can achieve this without k8s, though. If your goal is, "I want zero-downtime deploys," that alone is not sufficient reason to reach for something as massively complex as k8s. Set up a reverse proxy and do blue-green deploys behind it.
As I see it, managed Kubernetes basically gives me the same abstraction I’d have with Compose, except that I can add nodes easily, have some nice observability through GKE, etc. Compose might be simpler if I were running the cluster myself, but because GKE takes care of that, it’s one less thing that I have to do.
"Set up a reverse proxy and do blue-green deploys behind it."
I think this already introduces enough complexity and edge cases to make reinventing the wheel a bad idea. There's a lot involved in doing it robustly.
There are alternatives to Kubernetes (I prefer ECS/Fargate if you're on AWS), but trying to do it yourself to a production-ready standard sets you up for a lot of unnecessary yak shaving imho.
This sounds like terrible advice. Managing a reverse proxy with blue-green deploys behind it is not going to be trivial, and you have to roll most of that yourself. The deployment scripts alone are going to be hairy. Getting the same from K8s requires having a deploy.yaml file and a `kubectl apply -f <file>`. K8s is way less complex.
I ran such a system in prod over 7 years with >5-9s uptime, multiple deploys per day, and millions of users interacting with it. Our deploy scripts were ~10 line shell scripts, and any more complex logic (e.g. batching, parallelization, health checks) was done in a short Go program. Anyone could read and understand it in full. It deployed much faster than our equivalent stack on k8s.
k8s is a large and complex tool. Anyone who's run it in production at scale has had to deal with at least one severe outage caused by it.
It's an appropriate choice when you have a team of k8s operators full-time to manage it. It's not necessarily an appropriate choice when you want a zero-downtime deploy.
> It's an appropriate choice when you have a team of k8s operators full-time to manage it.
Are you talking about a full self-run type of scenario where you setup and administer k8s entirely yourself, or a managed system or semi-managed (like OpenShift)? Because if the former then I would agree with you, although I wouldn't recommend a full self-run unless you were a big enough corp to have said team. But if you're talking about even a managed service, I would have to disagree. I've been running for years on a managed service (as the only k8s admin) and have never had a severe outage caused by K8s
It isn’t, sadly, but the logic is straightforward. Have a set of IPs you target, iterate with your deploy script targeting each, check health before continuing. If anything doesn’t work (e.g. health check fails), stop the deploy to debug. There’s no automated rollback—simply `git revert` and run the deploy script again.
Did you manually promote deployments from one stage to another? This level of manual intervention is not sustainable if you deploy multiple times a day. How often did you deploy?
>> Managing a reverse proxy with blue-green deploys behind it is not going to be trivial, and you have to roll most of that yourself.
There are a lot of reverse proxies that will do this. Traditionally this was the job of a load balancer. With that being done by "software" you get the fun job of setting it up!
The hard part is doing it the first time, and having a sane strategy. What you want to do is identify and segment a portion of your traffic. Mostly this means injecting a cooking into the segmented traffics HTTP(S) requests. If you dont have a group of users consistently on the new service you get some odd behavior.
The deployment part is easy. Cause your running things concurrently then ports matter. Just have the alternate version deployed on a different port. This is not a big deal and is supper easy to do. In fact your deployments are probably set up to swap ports anyway. So all your doing is not committing to a final step in that process.
But... what if it is a service to service call inside your network. That too should be easy. Your passing id's around between calls for tracing right? Rather than "random cookie" you're just going to route based on these. Again easy to do in a reverse proxy, easier in a load balancer.
It's not like easy blue green deploys are some magic of kuberneties. We have been doing them for a long time. They were easy to do once set up (and highly scripted as a possible path for any normal deployment).
Kubernetes is to operations what rails is to programing... Its good, fast, helpful... till it isnt and then your left having buyers remorse.
Is there something like a k1s? What I’d love is “run this set of containers on this machine. If the machine goes down, I don’t care—I will fix it.” If it wired into nginx or caddy as well, so much the better. Something like that for homelab use would be wonderful.
You've basically described k3s, I think. I run it in my homelab (though I am enough of a tryhard to have multiple control planes) as well as on a couple of cloud servers as container runtimes (trading some overhead for consistency).
k3s really hammers home the "kubernetes is a set of behaviors, not a set of tools" stuff when you realize you can ditch etcd entirely and use sqlite if you really want to, and is a good learning environment.
Docker Compose probably fits the bill for that. They also have a built in minimalist orchestrator called Swarm if you do want to extend to multiple machines. I suppose it's considered "dead" since Kubernetes won mindshare, but it still gets updates.
Docker bare bones or docker compose. Run as systemd services and have docker run the container as a service account. Manual orchestration is all you need. Anything else like rancher or whatever are just fluff.
I run all my projects on Dokku. It’s a sweet spot for me between a barebones VPS with Docker Compose and something a lot more complicated like k8s. Dokku comes with a bunch of solid plugins for databases that handle backups and such. Zero downtime deploys, TLS cert management, reverse proxies, all out of the box. It’s simple enough to understand in a weekend and has been quietly maintained for many years. The only downside is it’s meant mostly for single server deployments, but I’ve never needed another server so far.
Just a note: Dokku has alternative scheduler plugins, the newest of which wraps k3s to give you the same experience you’ve always had with Dokku but across multiple servers.
Dokku really is a game changer for small business. It makes me look like a magician with deploys in < 2m (most of which is waiting for GitHub Actions to run the tests first!) and no downtime.
When people call Kubernetes a "great piece of technology", I find it the same as people saying the United States is the "greatest country in the world". Oh yeah? Great in what sense? Large? Absolutely. Powerful? Definitely. But then the adjectives sort of take a turn... Complicated? Expensive? Problematic? Threatening? A quagmire? You betcha.
If there were an alternative to Kubernetes that were just 10% less confusing, complicated, opaque, monolithic, clunky, etc, we would all be using it. But because Kubernetes exists, and everyone is using it, there's no point in trying to make an alternative. It would take years to reach feature parity, and until you do, you can't really switch away. It's like you're driving an 18-wheeler, and you think it kinda sucks, but you can't just buy and then drive a completely different 18 wheeler for only a couple of your deliveries.
You probably will end up using K8s at some point in the next 20 years. There's not really an alternative that makes sense. As much as it sucks, and as much as it makes some things both more complicated and harder, if you actually need everything it provides, it makes no sense to DIY, and there is no equivalent solution.
People forgot just how much of a mess Mesos environment was in comparison.
And often pushed Nomad to this day surprises me with randomly missing a feature or two that turns out to be impactful enough to want to deal with more complexity because ultimately the result was less complexity in total.
Sure, everyone has their own product and experience and it's fine to express it, but I don't get the usage of other decisions such as "no to services meshes", "no to helm" and many more.
You don't want to ideally reinvent the wheel for every workload you need (say you need a OIDC endpoint, an existing application): you are tempted to write everything from scratch by yourself, which is also fine, but the point is: why?
Many products deliver their own Helm package. And if you are sick of writing YAML, I would look for Terraform over Pulumi, for the reason that you use the same tool for bringing up Infrastructure and then workloads.
Kubernetes itself isn't easy to be used, in many cases you don't need it, but it might bring you nice things straight out of the box with less pain than other tooling (e.g. zero downtime deployments)
The problem with Helm is that it did the one thing you should not do, and refused to fix it even when their promised to.
They do text-replacement templating for YAML.
I have once spent a month, being quite experienced k8s wrangler, trying to figure out why Helm 2 was timing out, only to finally trace it down to how sometimes we would get wrong number of spaces in some lines.
I admit that I use some Helm stuff in my home environment, but for production I'm genuinely worried about the need to support whatever they've thrown into it. At minimum I'm going to have to study the chart and understand exactly what they propose to open-palm slam into my cluster, and for many/most applications at that point it might genuinely be worth just writing a manifest myself. Not always. Some applications are genuinely complex and need to be! But often, this has been the case for me. For all my stuff, though, I use kustomize and I'm pretty happy with it; it's too stupid for me to be clever, and this is good.
Service meshes are a different kettle of fish. hey add exciting new points of failure where they need not exist, and while there are definitely use cases for them, I'd default to avoiding them until somebody proves the need for one.
I honestly don't understand it either. Familiarity? K8s has like, what, 5 big concepts to know and once you are there the other concepts (generally) just build from there.
- Containers
- Pods
- Deployments
- Services
- Ingresses
There are certainly other concepts you can learn, but you aren't often dealing with them (just like you aren't dealing with them when working with something like docker compose).
Which is why you don't lose your SRE/OPS team just because you k8s.
I'd say that if you aren't big enough to have dedicated SRE then k8s is not for you. However, it really only takes 1 or 2 people to manage pretty large clusters with 100s or 1000s of deployments.
People don't understand k8s and are thus hating. K8s is a wonderful tool for most things many teams need. It may not be useful for homelab type of stuff as the learning curve is steep, but for professional use it cannot be beat currently. Just a bunch of I know what I'm doing and don't need this complicated thing I don't understand. Pretty simple and especially in a forum such as HN where we all are "experts" and need to explain to ourselves, and crucially others, why we are right not to use k8s. Bunch of children really.
Interesting that they avoid helm. It is the "plug and play" solution for Kubernetes. However, that is only in theory. My experience with most operators out there was clunky, buggy, or very limited and did not expose everything needed. But I still end up using helm itself with the combination of ArgoCD.
Helm is just a mess. If you're going to deploy something from helm, you're better off taking it apart and reconstructing it yourself, rather than depending on it to work like a package manager
In my experience, if you use first-party charts (= published by the same people that publish the packaged software) that are likely also provided to enterprise customers you'll have a good time (or at least a good starting point). For third-party charts, especially for more niche software I'd also rather avoid them.
I think the important detail here is that he mentions he doesn't use it because of operators. That may mean they tried it in previous major version which used teller. That was quite a long time ago.
That being said, helm templates are disgusting and I absolutely hate how easily developers complicate their charts. Even the default empty chart has helpers. Why, on Earth, why?
I almost fully relate to OPs aproach to k8s but I think with their simplified approach helm (the current one) could work quite well.
We avoided Helm as well. We found that Kustomize provides enough templating to cover almost all the common use cases and it's very easy for anyone to check their work, kubectl kustomize > compiled.yaml. FluxCD handles postbuild find and replace.
At most places, your cluster configuration is probably pretty set in stone and doesn't vary a ton.
There were some "hype cycles" (in Gartner's lingo) that I avoided during my career. The first one was the MongoDB/NoSQL hype - "Let's use NoSQL for everything!" trend. I tried it in a medium sized project and burnt my finger and it was right around when HN was flooded with full of "Why we migrated to MongoDB" stories.
The next one was Microservices. Everyone was doing something with microservices and I was just on a good 'ole Ruby on Rails monolith. Again, the HN stories came and went "Why we broke down our simple CRUD app into 534 microservices".
The final one was Kubernetes. I was a Cloud consultant in my past life and had to work with a lot of my peers who had the freedom to deploy In any architecture they saw fit. A bunch of them were on Kubernetes and I was just on a standard Compute VM for my clients.
We had a requirement from our management that all of us had to take some certification courses so they would be easily to pitch to clients. So, I prepped for one and read about Kubernetes and tried deploying a bunch of applications only to realize it was a very complex piece of moving parts - unnecessarily I may add. I was never able to understand why this was pushed on as normal. It made my decision to not use it only stronger.
Over the course of the 5 year journey, my peers' apps would randomly fail and they would be sometimes pulled over the weekends to push fixes to avert the P1 situation whilst I would be casually chilling in a bar with my friends. My compute engine VM, till date, to its credit has only had one P1 situation yet. And that was because the client forgot to renew their domain name.
Out of all the 3 hype cycles that I avoided in my career, the Kubernetes is the one I really am thankful of evading the most. This sort of complexity should not be normalised. I know this maybe unpopular opinion on HN, but I am willing to bite the bullet and save my time and my clients' money. So, thanks for the hater's guide. But, I prefer to remain one. I'd rather call a spade one.
Early on in the container hype cycle we decided to convert some of our services from VMs to ECS. It was easy to manage and the container build times were so much better than AMI build times.
Some time down the road we got acquired, and the company that acquired us ran their services in their own Kubernetes cluster.
When we were talking with their two person devops team about our architecture, I explained that we deployed some of our services on ECS. "Have you ever used it?" I asked them.
"No, thank goodness" one of them said jokingly.
By this time it was clear that Kubernetes had won and AWS was planning its managed Kubernetes offering. I assumed that after I became familiar with Kubernetes I'd feel the same way.
After a few months though it became clear that all these guys did was babysit their Kubernetes cluster. Upgrading it was a routine chore and every crisis they faced was related to some problem with the cluster.
Meanwhile our ECS deploys continued to be relatively hassle free. We didn't even have a devops team.
I grew to understand that managing Kubernetes was fun for them, despite the fact that it was overkill for their situation. They had architected for scale that didn't exist.
I felt much better about having chosen a technology that didn't "win".
So you don't use things you don't understand, valid point. But, saying others are using k8s as a way to use up free time is pretty useless too as we have managed k8s offerings and thus don't need the exercise. If you don't need k8s don't use it, thanks. Pretty useless story honestly
A lot depended on whether the ECS fit what you needed. ECSv1, even with FarGate, was so limited that my first k8s use was pretty much impossible on it at sensible price points, for example.
Something struck with me here that I've been thinking about. OP says a human should never wait for a pod. Agreed, it is annoying and sometimes means waiting for an EC2 and the pod.
We have jobs that users initiate that use 80+GB of memory and a few dozen cores. We run only one pod per node because the next size up EC2 costs a fortune and performance tops out on our current size.
These jobs are triggered via a button click that trigger a lambda that submits a job to the cluster. If it is a fresh node, user has to wait for the 1gb container to download from ECR. But it is the same container that the automated jobs that kick off every few minutes also uses, to rarely is there any waiting. But sometimes there is.
Should we be running some sort of clustering job scheduler that gets the job request and distributes work amongst long running pods in the cluster? My fear is that we just creat another layer of complexity and still end up waiting for the EC2, waiting for the pod to download, waiting for the agent now running on this pod to join the work distribution cluster.
However, we probably could be more proactive with this because we could spin up an extra pod+EC2 when the work cluster is 1:1 job:ec2.
Thoughts?
We're in the process of moving to Karpenter, so all this may be solved for us very soon with some clever configuration.
If you don't want to change the setup too much, consider running your nodes off an AMI with pre-loaded image. Maybe also ensure how exactly the images are layered, so if necessary you can reduce amount of "first boot patch" download.
There is a difference between waiting and waiting.
For an hourly batch job that already takes 10 minutes to run, the extra time for pod scheduling and container downloading is negligible anyway.
What you shouldn’t do is put pod scheduling in places where thousands of users per minute expect sub-second latency.
In your case, if the time for starting up the EC2 becomes a bigger factor than the job itself, you can add placeholder pods that just sleep, while requiring exactly that machine config but request 0 cpus, just to make sure it stays online.
I know it's fashionable to hate on Kubernetes these days, and it is overly complex and has plenty problems.
But what other solution allows you to:
* declarative define your infrastructure
* gives you load balancing, automatic recovery and scaling
* provides great observability into your whole stack (kubectl, k9s, ...)
* has a huge amount of pre-packaged software available (helm charts)
* and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
The answer is: there isn't any.
Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end.
You’re right that kubernetes is a bit batteries included, and for that its tempting to take it off the shelf because it “does a lot of needed things”, but you don’t need one tool to do all of those things.
It is ok to have domain specific processes or utilities to solve those.
You missed what I think is the most important point in OP's list: it does all of the above in a cloud agnostic way. If I want to move clouds with TF I'm rewriting everything to fit into a new cloud's paradigm. With Kubernetes there's a dozen providers built in (storage, loadbalancing, networking, auto scaling, etc.) or easy to pull in (certificates, KMS secrets, DNS); and they make moving clouds (and more importantly) running locally much easier.
Kubernetes is currently the best way to wrap up workloads in a cloud agnostic way. I've written dozens of services for K8s using different deployment mechanisms (Helm, Carvel's kapp, Flux, Kustomize) and I can run them just as easily in my home K8s cluster and in GCP. It's honestly incredible; I don't know of any other cloud tech that lets me do that.
One thing I think a lot of people miss too, is how good the concepts around Operators in Kubernetes are. It's hard to see unless you've written some yourself, but the theory around how operators work is very reminiscent of reactive coding in front end frameworks (or robotics closed loop control, what they were originally inspired by). When written well they're extremely resilient and incredibly powerful, and a lot of that power comes from etcd and the established patterns they're written with.
I think Kubernetes is really painful sometimes, and huge parts of it aren't great due to limitations of the language it's written in; but I also think it's the best thing available that I can run locally and in a cloud with a FOSS license.
> it does all of the above in a cloud agnostic way.
I'll give you the benefit of the doubt here and say that some of the basics are indeed cloud agnostic.
However, it's plain for many or most to see that outside of extremely "toy" workloads you will be learning a specific "flavour" of Kubernetes. EKS/GKE/AKS etc; They have, at minimum, custom resource definitions to handle a lot of things and at their worst have implementation specific (hidden) details between equivalent things (persistent volume claims on AWS vs GCP for example is quite substantially different).
For multicloud I usually think of my local K8s cluster and GKE, it's been a few years since I touched EKS. I'd love to hear your opinions on the substantive differences you run into. When switching between clouds I'm usually able to get away with only changing annotations on resources, which is easy enough to put in a values.yml file. I can't remember the last time I had to use a cloud specific CRD. What CRD's do you have to reach for commonly?
Thinking about it; the things I see as very cloud agnostic: Horizontal pod autoscaling, Node autoscaling, Layer 4 loadbalancing, Persistent volumes, Volume snapshots, Certificate managment, External DNS, External secrets, Ingress (when run in cluster, not through a cloud service),
That ends up covering a huge swath of my usecases, probably 80-90%. The main pain points I usually run into are: IAM, Trying to use cloud layer 7 ingress (app loadbalancers?)
I totally agree the underlying implementation if resources can be very different, but that's not the fault of Kubernetes; it's an issue with the implementation from the operator of the K8s cluster. All abstractions are going to be leaky at this level. But for PVCs I feel like storageclasses capture that well, and can be used to pick the level of performance you need per cloud; without having to rewrite the common provision of block device.
Something feels very off and mantra-like with the proportionality of how often cloud migration benefits are being presented as something very important to how often that actually happens in practice. Not to even mention that it also assumes that simpler setups are automatically harder to move around between clouds, or at least that there are a significant difference in required effort.
When I say it's easy to move between clouds, I'm not referring to an org needing to pick up everything and move from AWS to GCP. That is rare, and takes quite a bit of rearchitecting no matter what.
When I say something is easy to move, I mean that when I build on top of it, it's easy for users to run it in their cloud of choice with changes in config. It also means I have flexibility with where I choose to run something after I've developed it. For example I develop most stuff against minikube, then deploy it to GCP or a local production k8s. If I was using Terraform I couldn't do that.
Not sure what kind of apps this is but I can't see the big value-add on a golang app binary wrt to being cloud agnostic, nor wrt local development. It makes even less assumptions on the user's env. Still need some cloud conf (terraform, database etc) either way
If you'll excuse a slight digression, but I think there's a tendency atm to rather pay $1 in extra complexity a 100 times over time, than pay a $5 one-time fee. Like if repeating something similar twice - even if it's easy and not really a lot of effort - is a sign of failure and thus unbearable.
* your stack almost always ends up closely tied to one cloud provider. I've done and seen cloud migrations. They are so painful and costly that they often just aren't attempted.
* Cloud services make it much harder to run your stack locally and on CI. There are solutions and workarounds, but they are all painful. And you always end up tied to the behaviour of the particular cloud services
> but you don’t need one tool to do all of those things
To get the same experience, you do. And I don't see why you would want multiple tools.
If anything, Kubernetes isn't nearly integrated and full-featured enough, because it has too many pluggable parts leading to too much choice and interfacing complexity. Like pluggable ingress, pluggable state database, pluggable networking stack, no simple "end to end app" solution ( KNative, etc), ...
This overblown flexibility is what leads to most of the pain and perceived complexity, IMO.
Perhaps a little on the tinfoil hat side of things, but it isn't completely unreasonable to think that some of the FUD could originate from cloud providers. Kubernetes is a commoditizing force to some extent.
> This overblown flexibility is what leads to most of the pain and perceived complexity, IMO.
Huh, I guess you are spot on. My first experience with kubernetes was k3s and I couldn't for a long time figure out what's all the fuss is about and where is all that complexity people talk so much about. But then I tried vanilla kubernetes.
Far from it. TF is mostly writing static content, maybe read one or two things. It’s missing the runtime aspect of it, so are most cloud offerings, without excessive configuration. Rollouts, health probes, logs, service discovery. Just to name a few.
The thing is, "kubernetes" doesn't give you that either. You want a LB? Here's a list of them that you can add to a cluster. But actually pick multiple, because the one you picked in AWS doesn't support bare metal.
> because the one you picked in AWS doesn't support bare metal
That's just because AWS's Kubernetes offering is laughably bad.
There is huge difference in your experience whether you use Kubernetes via GKE (Autopilot) or any other solution (at least as long you don't have a dedicated infrastructure team).
I think "bare" Kubernetes is still a quite nice tool that allows for learning transferable skills across clouds (similar to Terraform). E.g. even if I have to spin up my own nginx-ingress to be able to handle ingress resources, after having learned that initially, I can basically do the same thing across clouds.
It's just that GKE (Autopilot) does a lot of those out of the box for you that, so you get a much easier end-to-end experience for non-admins (= "request resources -> have them instantiated").
Bare metal kubernetes is certainly a lot less complete out of the box when it comes to networking and storage but, people can, and often should, use a managed k8s service which provides all those things out of the box. And if you’re on bare metal once the infra team has abstracted away everything into LoadBalancers and StorageClasses it’s basically the same experience for end users of the cluster.
If you're talking about OpenShift on rented commodity compute, maybe. If you're talking about GKE/AKS/EKS or similar, I disagree wholeheartedly; you're then paying several multiples on the compute and a little extra for Kubernetes.
I like to think that most people who are upset at Kubernetes don't hate on all of it. I think the configuration aspect (YAML) and the very high level of abstraction is what get people lost and as a result they get frustrated by it. I've certainly fall in that category while trying to learn how to operate multiple clusters using different topologies and cloud providers.
But from an operational standpoint, when things are working, it usually behaves very well until you hit some rough edge cases (upgrades were much harder to achieve a couple of years back). But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.
> I like to think that most people who are upset at Kubernetes don't hate on all of it. I think the configuration aspect (YAML) …
I question the competence of anyone who does not question (and rag on) the prevalence of templating YAML.
> But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.
Damn straight. It’s only bad because everything else is strictly worse.
What are the reasons to not use JSON rather than YAML? From my admittedly-shallow experience with k8s, I have yet to encounter a situation in which I couldn't use JSON. Does this issue only pop up once you start using Helm charts?
at the surface level yaml is a lot easier to read and write for a human. less "s. but once you start using it for complex configuration it becomes unwieldy. but at that point json is also not better than yaml.
after using cdk i think that writing typescript to define infra is a significantly better experience
> and thus doesn't tie you into a specific cloud provider
It ties you to k8s instead, and it ties you to a few company wide heroes, and that is not a 'benefit' as it's being touted here.
Being tied to a cloud is not a horrible situation either. I suspect "being tied to a cloud" is a boogeyman that k8s proponents would like to spread, but just like with k8s, with the right choices, cloud integration is a huge benefit.
> * has a huge amount of pre-packaged software available (helm charts)
> * and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
NixOS. I have no clue about kubernetes, but I think NixOS even goes much deeper in these points (e.g. kubernetes is at the "application layer" and doesn't concern itself with declaratively managing the OS underneath, if I understand right). The other points seem much more situational, and if needed kubernetes might well be worth it. For something that could be a single server running a handful of services, NixOS is amazing.
I wouldn't say completely orthogonal. E.g. the points I've cited are overlap between the two, and ultimately both are meant to host some kind of services. But yes NixOS by itself manages a single machine (although combined with terraform it can become very convenient to also manage a fleet of NixOS machines). Kubernetes manages services on a cluster, but given how powerful a single machine can be I do think that many of those clusters could also just be one beefy server (and maybe a second one with some fail over mechanism, if needed).
If the cluster is indeed necessary though, I think NixOS can be a great base to stand up a Kubernetes cluster on top of.
There are lots of native NixOS tools for managing whole clusters (NixOps, Disnix, Colmena, deploy-rs, Morph, krops, Bento, ...). Lots of people deploy whole fleets of NixOS servers or clusters for specific applications without resorting to Kubernetes. (Kube integrations are also popular, though.) Some of those solutions are very old, too.
Disnix has been around for a long time, probably since before you ever heard of NixOS.
There is no easy solution to manage services and infrastructure: people who hate kubernetes complexity often underestimate the efforts of developing on your own all the features that k8s provides.
At the same time, people who suggest everyone to use kubernetes independently on the company maturity often forget how easy it is to run a service on a simple virtual machine.
In the multidimensional space that contains every software project, there is no hyperplane that separates when it’s worth to use kubernetes or not. It depends on the company, the employees, the culture, the business.
Of course there are general best practices, like for example if you’re just getting started with kubernetes, and already in the cloud, using a managed k8s service from your cloud provider could be a good idea. But again, even for this you’re going to find opposing views online.
> There is no easy solution to manage services and infrastructure: people who hate kubernetes complexity often underestimate the efforts of developing on your own all the features that k8s provides.
This. I was trying to create some infrastructure and application once, using various AWS and off the shelf components. I stopped halfway through when I realized I was reinventing k8s, very poorly. That's when I switched gears and learned k8s.
With that said, I use it sparingly due to the inherent complexity it brings, but at least I have a better handle on how and when it should be used and when it should not, and what problems it solved, since I myself was trying to solve some of the problems.
When I reflect what Netflix did back in 2010ish on AWS:
* The declarative infra is EC2/ASG configurations plus Jenkins configurations
* Client-side load balancing
* ASG for autoscaling and recovery
* Amazing observability with a home-grown monitoring system by 4 amazing engineers
Most of all, each of the above item was built and run by one or two people, except the observability stack with four. Oh, standing up a new region was truly a non-event. It just happened and as a member of the cloud platform team I couldn't even recall what I did for the project. It's not that Netflix's infra was better or worse than using k8s. I'm just amazed how happy I have been with an infra built more than 10 years ago, and how simple it was for end users. In that regard, I often question myself what I have missed in the whole movement of k8s platform engineering, other than people do need a robust solution to orchestrate containers.
There are an enormous number of tools that meet these requirements, most obviously Nomad. But really any competently-designed system, defined in terms of any cloud-agnostic provisioning system (Chef, Puppet, Salt, Ansible, home-grown scripts) would qualify.
And, for the record, observability is something very much unrelated to kubectl or k9s.
This is how I accomplished these things before. It involved simpler independent pieces which would not collapse whenever something went wrong. They were easy to reason about and building and fixing such tooling did not require me to hire an expensive consultant.
* declarative define your infrastructure
Declare in README.md that we have 3 web servers and that
Bob Jones set them up and manages them. Include Bob's email address and phone number.
* gives you load balancing, automatic recovery and scaling
Load balancing via a load balancer or another scheme. DNS is good enough for some cases. There are other solutions.
Automatic recovery - daemon scripts on the box to start all services when the box boots. VPS provider bounces the box when it crashes. That's one. There are others.
Scaling - automatic scaling is not needed at vast majority of companies that are starting out. When we need to scale to 4 servers, change README.md and send Bob Jones a quick message.
* provides great observability into your whole stack (kubectl, k9s, ...)
This is a need that is introduced because of k8s. There's a lot less to observe without k8s, and tools exist for it.
It's like saying "my backhoe has great diagnostic tools for diagnosing backhoe issues." That's true, but I don't have a backhoe and don't need a backhoe for what I am doing.
* has a huge amount of pre-packaged software available (helm charts)
This is a need that is introduced because of k8s. See above.
* allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
> doesn't tie you into a specific cloud provider
Not a real problem for most companies. If you're preparing to change cloud providers from day one, you are likely spending time on the wrong problem.
> same infrastructure in different envs
This is a benefit, which other solutions come close to, but k8s shines at. You can go a long way without having reproducible multi-machine setups in different envs and can come pretty close when needed, with manual work.
> Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end.
If true, this is a strange design choice. I'd be wary of anything that was made complex just for the sake of it.
k8s gets enough flack without having to accuse it of being complex just for funsies.
> But it's still by far the best we've got.
k8s is the best we've got when we want k8s. The trick is to not want k8s for the sake of wanting k8s.
There are times when k8s provides tremendous value. Most companies who decide to use it do not have the problems that k8s promises to solve, and never will. Sadly, sometimes it's because they've spent their time and money on unnecessary complexity like k8s instead of building a product that delivers value.
Good article. I used to be a k8s zealot (both CKAD and CKA certified) but have come to think that the good parts of k8s are the bare essentials (deployments, services, configmaps) and the rest should be left for exceptional circumstances.
Our team is happy to write raw YAML and use kustomize, because we prefer keeping the config plain and obvious, but we otherwise pretty much follow everything here.
> Hand-writing YAML. YAML has enough foot-guns that I avoid it as much as possible. Instead, our Kubernetes resource definitions are created from TypeScript with Pulumi.
LOL so, rather than linting YAML, bring in a whole programming language runtime plus third party library, adding yet another vendor lock, having to maintain versions, project compiling, moving away from K8S, adding mental overhead...
Managing structures in programming language is easier than dealing with finicky optional serialization format.
I have drastically reduced the amount of errors, mistakes, bugs, plain old wtf-induced hair pulling, by just mandating avoidance of YAML (and Helm) and using Jsonnet. Sure, there was some up-front work to write library code, but afterwards? I had people introduced to JSonnet with example deployment on one day, and shipping production-ready deployment for another app the next day.
We use Pulumi for IAC of non-k8s cloud resources too, so it doesn't introduce anything extra. In reality all but the smallest Kubernetes services will want something other than hand-written YAML: Helm-style templating, HCL, etc. TypeScript gives us type safety, and composable type safety. E.g. we have a function that encapsulates our best practices for speccing a deployment, and we get type safety for free across that function call boundary. Can't do that with YAML.
Most devops disaster stories I’ve heard lately are the result of endless addition of new tools. People join the company, see a problem, and then add another layer of tooling to address it, introducing new problems in the process. Then they leave the company, new people join, see the problems from that new tooling, add yet another layer of tooling, continuing the cycle.
I was talking to someone from a local startup a couple weeks ago who was trying to explain their devops stack. The number of different tools and platforms they were using was in the range of 50 different things, and they were asking for advice about how to integrate yet another thing to solve yet another self-inflicted problem.
It was as though they forgot what the goal was and started trying to collect as much experience with as many different tools as they could.
Would you believe that there is a company that is using cdk8s to handle its K8S configuration, and that such "infrastructure as code" repo ("infrastructure as code", this is the current hype) counts 76k YAML LoCs and 24k TypeScript LoCs to manage a bunch of Rails apps together with their related services? Like, some of such apps have less LoC.
yaml is objectively a bad language for complicated configurations, and once you add string formatting on top of it, you now have a complicated and shitty system, yay.
hopefully jsonnet or that apple thing will get more traction and popularlity.
k8s is really about you and if it makes sense for your use case. It’s not universally bad or universally good, and I don’t feel that there is a minimum team size required for it to make sense.
Managing k8s, for me at least, is a lot easier than juggling multiple servers with potentially different hardware, software, or whatever else. It’s rare that businesses will have machines that are all identical. Trying to keep adding machines to a pool that you manage manually and keep them running can be very messy and get out of control if you’re not on top of it.
k8s can also get out of control though it’s also easier to reason about and understand in this context. Eg you have eight machines of varying specs but all they really have installed is what’s required to run k8s, so you haven’t got as much divergence there. You can then use k8s to schedule work across them or ask questions about the machines.
We've found kubernetes to be surprisingly fragile, buggy, inflexible, and strictly imperative.
People make big claims but then it's not declarative enough to look up a resource or build a dependency tree and then your context deadline is exceeded.
276 comments
[ 2.4 ms ] story [ 285 ms ] threadFrom service teams that have done the migrations, the things I hear consistently though are:
- when a Helm deploy fails, finding the reason why is a PITA (we run with --atomic so it'll rollback on a failed deploy. What failed? Was it bad code causing a pod to crash loop? Failed k8s resource create? who knows! have fun finding out!)
- they have to learn a whole new way of operating, particularly around in-the-moment scaling. A team today can go into the AWS Console at 4am during an incident and change the ASG scaling targets, but to do that with a service running in Kubernetes means making sure they have kubectl (and it's deps, for us that's aws-cli) installed and configured, AND remembering the `kubectl scale deployment X --replicas X` syntax.
[Both of those things are very much fixable]
This is different from waking people up at 4am frequently to bump up the number of replicas.
We've had pretty good success with simple HPAs.
I've been really happy with just using `envsubst` and environment variables to generate a manifest at deploy time. It's easy with most CI systems to "archive" the manifest, and it can then be easily read by a human or downloaded/applied manually for debugging with. Deploys are also just `cat k8s/${ENV}/deploy.yaml | envsubt > output.yaml && kubectl apply -f output.yaml`
I've also experimented with using terraform. It's actually been a good enough experience that I may go fully with terraform on a new project and see how it goes.
I cannot recommend terraform. I use it daily, and daily I wish I did not. I think Pulumi is the future. Not as battle tested, but terraform is a mountain of bugs anyway, so it can't possibly be worse.
Just one example where terraform sucks: You cannot both deploy a kubernetes cluster (say an EKS/AKS cluster) and then use kubernetes_manifest provider in a single workspace. You must do this across two separate terraform runs.
The problem is that they're custom and homegrown. Your organization alone invests in them, trains new staff in them, is responsible for debugging and fixing when they break, has to re-invest when they no longer do all the things you want. DIY frameworks ultimately end up as byzantine and labyrinthine as Kubernetes itself. The virtue of industry platforms like Kubernetes is, however complex and only half-baked they start, over time the entire industry trains on them, invests in them, refines and improves them. They benefit from a long-term economic virtuous cycle that DIY rarely if ever can. Even the longest, strongest, best-funded holdouts for bespoke languages, OSs, and frameworks—aerospace, finance, miltech—have largely come 'round to COTS first and foremost.
The whole thing is a bolt-on that has to spend a ton of time working around the limitations of the foundation, and it shows.
Unfortunately there seems to be zero interest in fixing that and so much sunk cost in existing Unix/Posix designs that it seems like we are completely stuck with a basic foundation of outdated brittleness.
What I think we need:
* An OS that runs hardware-independent code (WASM?) natively and permits things like hot updates, state saving and restoration, etc. Abstract away the hardware.
* Native built-in support for clustering, hot backups, live process migration between nodes, and generally treating hardware as a pure commodity in a RAIN (redundant array of inexpensive nodes) configuration.
* A modern I/O API. Posix I/O APIs are awful. They could be supported for backward compatibility via a compatibility library.
* Native built-in support for distributed clustered storage with high availability. Basically a low or zero config equivalent of Ceph or similar built into the OS as a first class citizen.
* Immutable OS that installs almost instantly on hardware, can be provisioned entirely with code, and where apps/services can be added and removed with no "OS rot." The concept of installing software "on" the OS needs to be killed with fire.
* Shared distributed network stack where multiple machines can have the same virtual network interfaces, IPs, and open TCP connections can migrate. Built-in load balancing.
I'm sure people around here can think of more ideas that belong in this list. These are not fringe things that are impossible to build.
Basically you should have an immutable image OS that turns many boxes into one box and you don't have to think about it. Storage is automatically clustered. Processes automatically restart or, if a hardware fault is detected in time, automatically migrate.
There were efforts to build such things (Mosix, Plan 9, etc.) but they were bulldozed by the viral spread of free Unix-like OSes that were "good enough."
Edit:
That being said, I'm not saying Kubernetes is good software either. The core engine is actually decent and as the OP said has a lot of complexity that's needed to support what it does. The ugly nasty disgusting parts are the config interface, clunky shit like YAML, and how generally arcane and unapproachable and ugly the thing is to actually use.
I just loathe software like this. I feel the same way about Postgres and Systemd. "Algorithmically" they are fine, but the interface and the way you use them is arcane and makes me feel like I'm using a 70s mainframe on a green VT220 monitor.
Either these things are designed by the sorts of "hackers" who like complexity and arcane-ness, or they're hacks that went viral and matured into global infrastructure without planning. I think it's a mix of both... though in the case of Postgres it's also that the project is legitimately old. It feels like old-school Unix clunkware because it is.
[Edit] Parent comment is almost entirety completely different after that edit to what I responded to. But I think my point still stands. One day, hopefully in my lifetime, we shall see it.
The problem is that right now this gets you lock-in to a proprietary cloud. There are some loose standards but the devil's in the details and once you are deployed somewhere it's damn hard to impossible to move without serious downtime and fixing.
I can’t say I know it myself. It always looks good on paper. Strangely nobody uses it. There must be a catch that detracts from it?
Hopefully we should get OpenFaas and Lambda, in the same way we have ECS and EKS. Standardised ways to complete tasks, rather than managing imaginary servers.
We are still early in the cycle.
If CoreOS actually wanted to make distributed computing easier, they'd make patches for the Linux kernel (or make an entirely different kernel). See the many distributed OS kernels that were made over 20 years ago. But that's a lot of work. So instead they tried to go the cheap and easy route. But the cheap and easy route ends up being much shittier.
There's no commercial advantage to building a distributed OS, which is why no distributed OS is successful today. You would need a crazy person to work for 10 years on a pet project until it's feature-complete, and then all of a sudden everyone would want to use it. But until it's complete, nobody would use it, and nobody would spend time developing it. Even once it's created, if it's not popular, still nobody will use it (you can use Plan9 today, but nobody does).
https://en.wikipedia.org/wiki/Distributed_operating_system
Maybe not, but I'm confident that the system you're describing is impossible to build in a way that is both general and efficient.
If I have a block of storage that boots on one system, it should boot on another. An OS should poll for available services, allow them to define APIs and be utilized, but no bloat should be added to a base to support elective services.
Everything should be microkernel too, but now I’m just venting.
k8s's worst property is that it's a cleverness trap. You can do anything in k8s whether it's sane to do so or not. The biggest guardrail against falling into is managing your k8s with terraform-ish so that you don't find yourself in a spot where "effort to do it right" >> "effort to hack in YAML" and finding your k8s cluster becoming spaghetti.
Re: cleverness trap. I feel like this is the tragedy of software development. We like to be seen as clever. We are doing "hard" things. I have way more respect for engineers that do "simple" things that just work using boring tech and factor in whole lifecycle of the product.
Not everyone has money to burn, even back in ZIRP era.
And before you trot out wages for experienced operations team - I've regularly dealt with it being cheaper to pay for one or two very experienced people than deal with AWS bill.
For the very simple reason that cloud provider's prices are scaled to US market and not everyone has US money levels.
I wouldn't start with k8s and instead opt for ASGs until you reach the point where you look at your AWS account and see a bunch of EC2 instances sitting underutilized.
Is there a clear example of this? E.g. is kubernetes inherently unable to start a pod (assuming the same sequence of events, e.g. warm/cold image with streaming enabled) under 500ms, 1s etc?
I am asking this as someone who spent quite a bit of time and wasn't able to bring it down 2s< mark, which eventually led us to rewrite the latency sensitive parts to use Nomad. But we are currently in a state where we are re-considering kubernetes for its auxilary tooling benefits and would love to learn more if anyone had experiences with starting and stopping thousands of pods with the lowest possible latencies without caring for utilization or placement but just observable boot latencies.
a) preload all images of course
b) there's enough of nodes with enough capacity
c) the pods don't use anything that has possible longer latency (high latency CSI etc.)
d) you might want to write custom scheduler for your workloads (it could take into account what images are preloaded where, etc)
Big possible win is custom scheduling, but barely anyone seems to know it exists
(I made up the dozen number, but my point is that that design would be perfectly acceptable given Kubernetes' design constraints)
Something like Knative can allow for faster startup times if you follow the common best-practices (pre-fetching images, etc.), but I'm not sure if it supports enough of the session-related feature that you were probably looking for to be a stand-in for Plane.
But the article trending here a couple of days ago describes it well. https://www.theolognion.com/p/company-forgets-why-they-exist...
It is complex enough to make the k8s maintainers the heroes of the company. And this is where things tend to go sideways.
It has enough knobs and levers to distract the project from what they are actually trying to achieve.
Kubernetes is designed to solve big problems and if you don't have those problems, you're introducing a tonne of complexity for very little benefit. An ideal orchestrator would be more composable and not introduce more complexity than needed for the scale you're running at. I'd really like to see a modern alternative to K8S that learns from some of its mistakes.
You cannot pretend k8s doesn't exist in a k8s system.
If you are already using the cloud, maybe leverage abstraction already available in that context.
Vanilla Kubernetes is just enough abstraction to avoid both of those situations.
It can be cheaper to depend on cloud provider to ship some features, but with tools like crossplane you can abstract that out so developers can just "order" a database service etc. for their application.
None of that is prevented with SLA
Arguably if you can’t evaluate the raw cloud offerings and jump on a supposed silver bullet you need to stop immediately.
And with smaller companies I tend to find k8s way more cost effective. I pulled things I wouldn't be able to fit in a budget otherwise.
A few months later I transitioned the team to use containers with proper CI/CD and EKS with Terraform and Argo CD. The team and also the managers like it, since we could deploy quite quickly.
Our K8s clusters never goes more than a couple days without some sort of strange issue popping up. Arguably it could be because my company outsourced maintenance of it to an army of idiots. But K8s is a tool that is only as good as the operator, and competence can be hard to come by at some companies.
So I say unless you're at a company that pays top salaries for the top 5% of engineering talent, you're probably better off just using the AWS provided service.
Depending on your local market, AWS bills might be way worse than the cost of few bright ops people who will let you choose from offerings including running dev envs on random assortment of dedicated servers and local e-waste escapees
For larger scale orchestratiom, Hashicorp Nomad can also be a notable contender, while in some ways still being simpler than Kubernetes.
And even when it comes to Kubernetes, distros like K3s and tools like Portainer or Rancher can keep managing the cluster easy.
https://github.com/BretFisher/ama/discussions/146
To be honest I hardly see any reasonable/actionable advice from Cloud/SAAS vendors. Either it is to sell their stuff or generic stuff like "One should be securing / monitoring their stuff running in prod". Oh wow, never thought or done any such thing before.
Choosing k8s is not just based on scaling requirements anymore. There are also benefits of being compatible with a rich ecosystem of software.
It's surely possible to cobble all of this together without k8s, but k8s' main advantage is exposing a standardized API that simplifies managing this entire ecosystem. It often makes it worth the additional overhead of adopting, understanding and managing k8s itself.
How many people out there really need C# or object oriented programming?
The argument you present might be valid if you decide to use a tech stack prior having much experience with it.
K8s is remarkably easier to retain institutional knowledge as well as spread it.
Also, in my experience, you either have to spend ridiculous amounts of money on SaaS/PaaS, or you find that you have to host a lot more than just your application and suddenly the deployment story becomes more complex.
Depending on where you are and how much you're willing to burn money, you might find out that k8s experts are cheaper than the money saved by not going PaaS.
Why would anyone expect it? It's not their job, is it? We don't expect backend devs to know frontend and vice-versa, or any of them to have AWS certification. Why would it be different with k8s?
> Just chucking a container on a non-k8s managed platform (e.g. Cloud Run) would be much simpler, and no pile of bash scripts.
Simpler to deploy, sure, but not to actually run it seriously in the long term. Though, if we are talking about A container (as in singular), k8s would indeed be some serious over-engineering
To add insult to injury, I've seen more than one use IaC cloud tooling as an install script vs a maintainable and idempotent solution. It's all quite sad really.
The former is a very small set involving having huge amounts of bare metal systems.
The latter is suprisingly large set of companies, sometimes even with one server.
For most use case k8s is not there to give you HA but to give you a standard way of deploying a stack, that being on the cloud or on prem.
What's wonderful is that when I work on multiple clouds, my knowledge transfers just fine. I don't think of the AWS solution or the GCS solution, I use the same kubectl to check out both, view logs, inspect and fix.
Even when I got tired of waiting for GKE to spin up a node, running Github actions on a self-hosted microk8s meant instant pod starts and very little fuss. But using Kubernetes meant I got to take advantage of the Github operator, which let me reuse the same machine for multiple builds without the headaches.
When I want to run some open source, I often find a helm chart the helps me get set up. Nowadays running open source packages can involve all kinds of dependencies, but getting it running on a k8s cluster to check it out, or even in prod, is a relatively straight forward editing of some values files. I've recently ran Uptrace and Superset that way. They're not a bajillion requests per second setups, they don't have to be, and it was far easier to set up than most methods.
I would say your friend is right. Few people _need_ k8s but it's one interface to a bunch of complicated proprietary stuff. I can know core small, core set of k8s tools really well and forget half of the junk that I ever knew about public clouds. It's all the same patterns, transferable and reliable.
This is the key point. Even getting to the point where I could install Kubernetes myself on my own hardware took weeks, just understanding what hardware was needed and which of the (far too many) different installers I had to use.
Some technologies (like Kubernetes) tend to attract discussions where half of the commenters completely ignore the original article, so we end up having a weekly thread about Kubernetes where the points of the article (which are interesting) are not able to be discussed because they are drowned out by the same unstructured OT discussions?
At the time of this posting there are ~20 comments with ~2 actually having anything to do with the points of the article rather than Kubernetes in general.
Discussions of k8s pitfalls and successes in general seems to be very much in line with what the article is advocating. And, to that point, there's frankly just not a whole lot interesting in this article for discussion "We avoid yaml and operators"... Neat.
Yeah, and I think that provides a good basis to discussion, where people can critique/discuss whether the evaluation that the author has made are correct (which a few comments are doing). At the same time a lot of that discussion is being displaced by what I would roughly characterize as "general technology flaming" which isn't going anywhere productive.
I am rather happy that people are having general purpose discussion about K8s.
Kubernetes might be the right tool for the job if we accept that this is a necessary evil. But maybe it's not? The idea that I might fail to collaborate with you because a third party failed because a fourth party failed kind of smells like a recipe for software that breaks all the time.
That's what I currently use Kubernetes for. What stack are you proposing instead?
If you need to replicate storage, share networks and otherwise share resources across multiple hosts, kubernetes is better suited.
But you'll also have much less control with compose, e.g. no limiting of egress/ingress and more.
I think this already introduces enough complexity and edge cases to make reinventing the wheel a bad idea. There's a lot involved in doing it robustly.
There are alternatives to Kubernetes (I prefer ECS/Fargate if you're on AWS), but trying to do it yourself to a production-ready standard sets you up for a lot of unnecessary yak shaving imho.
k8s is a large and complex tool. Anyone who's run it in production at scale has had to deal with at least one severe outage caused by it.
It's an appropriate choice when you have a team of k8s operators full-time to manage it. It's not necessarily an appropriate choice when you want a zero-downtime deploy.
Are you talking about a full self-run type of scenario where you setup and administer k8s entirely yourself, or a managed system or semi-managed (like OpenShift)? Because if the former then I would agree with you, although I wouldn't recommend a full self-run unless you were a big enough corp to have said team. But if you're talking about even a managed service, I would have to disagree. I've been running for years on a managed service (as the only k8s admin) and have never had a severe outage caused by K8s
There are a lot of reverse proxies that will do this. Traditionally this was the job of a load balancer. With that being done by "software" you get the fun job of setting it up!
The hard part is doing it the first time, and having a sane strategy. What you want to do is identify and segment a portion of your traffic. Mostly this means injecting a cooking into the segmented traffics HTTP(S) requests. If you dont have a group of users consistently on the new service you get some odd behavior.
The deployment part is easy. Cause your running things concurrently then ports matter. Just have the alternate version deployed on a different port. This is not a big deal and is supper easy to do. In fact your deployments are probably set up to swap ports anyway. So all your doing is not committing to a final step in that process.
But... what if it is a service to service call inside your network. That too should be easy. Your passing id's around between calls for tracing right? Rather than "random cookie" you're just going to route based on these. Again easy to do in a reverse proxy, easier in a load balancer.
It's not like easy blue green deploys are some magic of kuberneties. We have been doing them for a long time. They were easy to do once set up (and highly scripted as a possible path for any normal deployment).
Kubernetes is to operations what rails is to programing... Its good, fast, helpful... till it isnt and then your left having buyers remorse.
k3s really hammers home the "kubernetes is a set of behaviors, not a set of tools" stuff when you realize you can ditch etcd entirely and use sqlite if you really want to, and is a good learning environment.
If you want something crazy all-in-one for homelab check out https://github.com/azukaar/Cosmos-Server
For something more production oriented https://github.com/basecamp/kamal
https://dokku.com/
If there were an alternative to Kubernetes that were just 10% less confusing, complicated, opaque, monolithic, clunky, etc, we would all be using it. But because Kubernetes exists, and everyone is using it, there's no point in trying to make an alternative. It would take years to reach feature parity, and until you do, you can't really switch away. It's like you're driving an 18-wheeler, and you think it kinda sucks, but you can't just buy and then drive a completely different 18 wheeler for only a couple of your deliveries.
You probably will end up using K8s at some point in the next 20 years. There's not really an alternative that makes sense. As much as it sucks, and as much as it makes some things both more complicated and harder, if you actually need everything it provides, it makes no sense to DIY, and there is no equivalent solution.
And often pushed Nomad to this day surprises me with randomly missing a feature or two that turns out to be impactful enough to want to deal with more complexity because ultimately the result was less complexity in total.
Sure, everyone has their own product and experience and it's fine to express it, but I don't get the usage of other decisions such as "no to services meshes", "no to helm" and many more.
You don't want to ideally reinvent the wheel for every workload you need (say you need a OIDC endpoint, an existing application): you are tempted to write everything from scratch by yourself, which is also fine, but the point is: why?
Many products deliver their own Helm package. And if you are sick of writing YAML, I would look for Terraform over Pulumi, for the reason that you use the same tool for bringing up Infrastructure and then workloads.
Kubernetes itself isn't easy to be used, in many cases you don't need it, but it might bring you nice things straight out of the box with less pain than other tooling (e.g. zero downtime deployments)
They do text-replacement templating for YAML.
I have once spent a month, being quite experienced k8s wrangler, trying to figure out why Helm 2 was timing out, only to finally trace it down to how sometimes we would get wrong number of spaces in some lines.
Service meshes are a different kettle of fish. hey add exciting new points of failure where they need not exist, and while there are definitely use cases for them, I'd default to avoiding them until somebody proves the need for one.
- Containers
- Pods
- Deployments
- Services
- Ingresses
There are certainly other concepts you can learn, but you aren't often dealing with them (just like you aren't dealing with them when working with something like docker compose).
Haven't had an issue once I fixed sizing.
I'd say that if you aren't big enough to have dedicated SRE then k8s is not for you. However, it really only takes 1 or 2 people to manage pretty large clusters with 100s or 1000s of deployments.
That's where my company currently finds itself.
What are the best resources to learn simple k8 in 2024?
That being said, helm templates are disgusting and I absolutely hate how easily developers complicate their charts. Even the default empty chart has helpers. Why, on Earth, why?
I almost fully relate to OPs aproach to k8s but I think with their simplified approach helm (the current one) could work quite well.
At most places, your cluster configuration is probably pretty set in stone and doesn't vary a ton.
The next one was Microservices. Everyone was doing something with microservices and I was just on a good 'ole Ruby on Rails monolith. Again, the HN stories came and went "Why we broke down our simple CRUD app into 534 microservices".
The final one was Kubernetes. I was a Cloud consultant in my past life and had to work with a lot of my peers who had the freedom to deploy In any architecture they saw fit. A bunch of them were on Kubernetes and I was just on a standard Compute VM for my clients.
We had a requirement from our management that all of us had to take some certification courses so they would be easily to pitch to clients. So, I prepped for one and read about Kubernetes and tried deploying a bunch of applications only to realize it was a very complex piece of moving parts - unnecessarily I may add. I was never able to understand why this was pushed on as normal. It made my decision to not use it only stronger.
Over the course of the 5 year journey, my peers' apps would randomly fail and they would be sometimes pulled over the weekends to push fixes to avert the P1 situation whilst I would be casually chilling in a bar with my friends. My compute engine VM, till date, to its credit has only had one P1 situation yet. And that was because the client forgot to renew their domain name.
Out of all the 3 hype cycles that I avoided in my career, the Kubernetes is the one I really am thankful of evading the most. This sort of complexity should not be normalised. I know this maybe unpopular opinion on HN, but I am willing to bite the bullet and save my time and my clients' money. So, thanks for the hater's guide. But, I prefer to remain one. I'd rather call a spade one.
Some time down the road we got acquired, and the company that acquired us ran their services in their own Kubernetes cluster.
When we were talking with their two person devops team about our architecture, I explained that we deployed some of our services on ECS. "Have you ever used it?" I asked them.
"No, thank goodness" one of them said jokingly.
By this time it was clear that Kubernetes had won and AWS was planning its managed Kubernetes offering. I assumed that after I became familiar with Kubernetes I'd feel the same way.
After a few months though it became clear that all these guys did was babysit their Kubernetes cluster. Upgrading it was a routine chore and every crisis they faced was related to some problem with the cluster.
Meanwhile our ECS deploys continued to be relatively hassle free. We didn't even have a devops team.
I grew to understand that managing Kubernetes was fun for them, despite the fact that it was overkill for their situation. They had architected for scale that didn't exist.
I felt much better about having chosen a technology that didn't "win".
We have jobs that users initiate that use 80+GB of memory and a few dozen cores. We run only one pod per node because the next size up EC2 costs a fortune and performance tops out on our current size.
These jobs are triggered via a button click that trigger a lambda that submits a job to the cluster. If it is a fresh node, user has to wait for the 1gb container to download from ECR. But it is the same container that the automated jobs that kick off every few minutes also uses, to rarely is there any waiting. But sometimes there is.
Should we be running some sort of clustering job scheduler that gets the job request and distributes work amongst long running pods in the cluster? My fear is that we just creat another layer of complexity and still end up waiting for the EC2, waiting for the pod to download, waiting for the agent now running on this pod to join the work distribution cluster.
However, we probably could be more proactive with this because we could spin up an extra pod+EC2 when the work cluster is 1:1 job:ec2.
Thoughts?
We're in the process of moving to Karpenter, so all this may be solved for us very soon with some clever configuration.
For an hourly batch job that already takes 10 minutes to run, the extra time for pod scheduling and container downloading is negligible anyway.
What you shouldn’t do is put pod scheduling in places where thousands of users per minute expect sub-second latency.
In your case, if the time for starting up the EC2 becomes a bigger factor than the job itself, you can add placeholder pods that just sleep, while requiring exactly that machine config but request 0 cpus, just to make sure it stays online.
But what other solution allows you to:
* declarative define your infrastructure
* gives you load balancing, automatic recovery and scaling
* provides great observability into your whole stack (kubectl, k9s, ...)
* has a huge amount of pre-packaged software available (helm charts)
* and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
The answer is: there isn't any.
Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end.
But it's still by far the best we've got.
You’re right that kubernetes is a bit batteries included, and for that its tempting to take it off the shelf because it “does a lot of needed things”, but you don’t need one tool to do all of those things.
It is ok to have domain specific processes or utilities to solve those.
Kubernetes is currently the best way to wrap up workloads in a cloud agnostic way. I've written dozens of services for K8s using different deployment mechanisms (Helm, Carvel's kapp, Flux, Kustomize) and I can run them just as easily in my home K8s cluster and in GCP. It's honestly incredible; I don't know of any other cloud tech that lets me do that.
One thing I think a lot of people miss too, is how good the concepts around Operators in Kubernetes are. It's hard to see unless you've written some yourself, but the theory around how operators work is very reminiscent of reactive coding in front end frameworks (or robotics closed loop control, what they were originally inspired by). When written well they're extremely resilient and incredibly powerful, and a lot of that power comes from etcd and the established patterns they're written with.
I think Kubernetes is really painful sometimes, and huge parts of it aren't great due to limitations of the language it's written in; but I also think it's the best thing available that I can run locally and in a cloud with a FOSS license.
I'll give you the benefit of the doubt here and say that some of the basics are indeed cloud agnostic.
However, it's plain for many or most to see that outside of extremely "toy" workloads you will be learning a specific "flavour" of Kubernetes. EKS/GKE/AKS etc; They have, at minimum, custom resource definitions to handle a lot of things and at their worst have implementation specific (hidden) details between equivalent things (persistent volume claims on AWS vs GCP for example is quite substantially different).
Thinking about it; the things I see as very cloud agnostic: Horizontal pod autoscaling, Node autoscaling, Layer 4 loadbalancing, Persistent volumes, Volume snapshots, Certificate managment, External DNS, External secrets, Ingress (when run in cluster, not through a cloud service),
That ends up covering a huge swath of my usecases, probably 80-90%. The main pain points I usually run into are: IAM, Trying to use cloud layer 7 ingress (app loadbalancers?)
I totally agree the underlying implementation if resources can be very different, but that's not the fault of Kubernetes; it's an issue with the implementation from the operator of the K8s cluster. All abstractions are going to be leaky at this level. But for PVCs I feel like storageclasses capture that well, and can be used to pick the level of performance you need per cloud; without having to rewrite the common provision of block device.
When I say something is easy to move, I mean that when I build on top of it, it's easy for users to run it in their cloud of choice with changes in config. It also means I have flexibility with where I choose to run something after I've developed it. For example I develop most stuff against minikube, then deploy it to GCP or a local production k8s. If I was using Terraform I couldn't do that.
If you'll excuse a slight digression, but I think there's a tendency atm to rather pay $1 in extra complexity a 100 times over time, than pay a $5 one-time fee. Like if repeating something similar twice - even if it's easy and not really a lot of effort - is a sign of failure and thus unbearable.
* your stack almost always ends up closely tied to one cloud provider. I've done and seen cloud migrations. They are so painful and costly that they often just aren't attempted.
* Cloud services make it much harder to run your stack locally and on CI. There are solutions and workarounds, but they are all painful. And you always end up tied to the behaviour of the particular cloud services
> but you don’t need one tool to do all of those things
To get the same experience, you do. And I don't see why you would want multiple tools.
If anything, Kubernetes isn't nearly integrated and full-featured enough, because it has too many pluggable parts leading to too much choice and interfacing complexity. Like pluggable ingress, pluggable state database, pluggable networking stack, no simple "end to end app" solution ( KNative, etc), ... This overblown flexibility is what leads to most of the pain and perceived complexity, IMO.
Huh, I guess you are spot on. My first experience with kubernetes was k3s and I couldn't for a long time figure out what's all the fuss is about and where is all that complexity people talk so much about. But then I tried vanilla kubernetes.
Docker Swarm has all those features for example.
(Not that I am recommending Docker Swarm.)
That’s a joke.
That's just because AWS's Kubernetes offering is laughably bad.
There is huge difference in your experience whether you use Kubernetes via GKE (Autopilot) or any other solution (at least as long you don't have a dedicated infrastructure team).
It's just that GKE (Autopilot) does a lot of those out of the box for you that, so you get a much easier end-to-end experience for non-admins (= "request resources -> have them instantiated").
Kidding of course. If you need anything approximating Kubernetes, use it. If you just need one machine maybe don't.
... when the problem was lacking config management. They overshot; use the right tool for the job
But from an operational standpoint, when things are working, it usually behaves very well until you hit some rough edge cases (upgrades were much harder to achieve a couple of years back). But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.
I question the competence of anyone who does not question (and rag on) the prevalence of templating YAML.
> But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.
Damn straight. It’s only bad because everything else is strictly worse.
So yaml formatters break it, humans struggle to generate code with proper indents, and it’s an insane mess. It’s horrendous.
What are the reasons to not use JSON rather than YAML? From my admittedly-shallow experience with k8s, I have yet to encounter a situation in which I couldn't use JSON. Does this issue only pop up once you start using Helm charts?
after using cdk i think that writing typescript to define infra is a significantly better experience
It ties you to k8s instead, and it ties you to a few company wide heroes, and that is not a 'benefit' as it's being touted here.
Being tied to a cloud is not a horrible situation either. I suspect "being tied to a cloud" is a boogeyman that k8s proponents would like to spread, but just like with k8s, with the right choices, cloud integration is a huge benefit.
> [...]
> * has a huge amount of pre-packaged software available (helm charts)
> * and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
NixOS. I have no clue about kubernetes, but I think NixOS even goes much deeper in these points (e.g. kubernetes is at the "application layer" and doesn't concern itself with declaratively managing the OS underneath, if I understand right). The other points seem much more situational, and if needed kubernetes might well be worth it. For something that could be a single server running a handful of services, NixOS is amazing.
Kubernetes manages a cluster, NixOS manages a single machine.
If the cluster is indeed necessary though, I think NixOS can be a great base to stand up a Kubernetes cluster on top of.
Disnix has been around for a long time, probably since before you ever heard of NixOS.
At the same time, people who suggest everyone to use kubernetes independently on the company maturity often forget how easy it is to run a service on a simple virtual machine.
In the multidimensional space that contains every software project, there is no hyperplane that separates when it’s worth to use kubernetes or not. It depends on the company, the employees, the culture, the business.
Of course there are general best practices, like for example if you’re just getting started with kubernetes, and already in the cloud, using a managed k8s service from your cloud provider could be a good idea. But again, even for this you’re going to find opposing views online.
This. I was trying to create some infrastructure and application once, using various AWS and off the shelf components. I stopped halfway through when I realized I was reinventing k8s, very poorly. That's when I switched gears and learned k8s.
With that said, I use it sparingly due to the inherent complexity it brings, but at least I have a better handle on how and when it should be used and when it should not, and what problems it solved, since I myself was trying to solve some of the problems.
* The declarative infra is EC2/ASG configurations plus Jenkins configurations * Client-side load balancing * ASG for autoscaling and recovery * Amazing observability with a home-grown monitoring system by 4 amazing engineers
Most of all, each of the above item was built and run by one or two people, except the observability stack with four. Oh, standing up a new region was truly a non-event. It just happened and as a member of the cloud platform team I couldn't even recall what I did for the project. It's not that Netflix's infra was better or worse than using k8s. I'm just amazed how happy I have been with an infra built more than 10 years ago, and how simple it was for end users. In that regard, I often question myself what I have missed in the whole movement of k8s platform engineering, other than people do need a robust solution to orchestrate containers.
Or at least that's how I got into k8s, because it allowed me to ship for 1/10th the price of my competitor.
And, for the record, observability is something very much unrelated to kubectl or k9s.
* declarative define your infrastructure
Declare in README.md that we have 3 web servers and that Bob Jones set them up and manages them. Include Bob's email address and phone number.
* gives you load balancing, automatic recovery and scaling
Load balancing via a load balancer or another scheme. DNS is good enough for some cases. There are other solutions.
Automatic recovery - daemon scripts on the box to start all services when the box boots. VPS provider bounces the box when it crashes. That's one. There are others.
Scaling - automatic scaling is not needed at vast majority of companies that are starting out. When we need to scale to 4 servers, change README.md and send Bob Jones a quick message.
* provides great observability into your whole stack (kubectl, k9s, ...)
This is a need that is introduced because of k8s. There's a lot less to observe without k8s, and tools exist for it.
It's like saying "my backhoe has great diagnostic tools for diagnosing backhoe issues." That's true, but I don't have a backhoe and don't need a backhoe for what I am doing.
* has a huge amount of pre-packaged software available (helm charts)
This is a need that is introduced because of k8s. See above.
* allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
> doesn't tie you into a specific cloud provider
Not a real problem for most companies. If you're preparing to change cloud providers from day one, you are likely spending time on the wrong problem.
> same infrastructure in different envs
This is a benefit, which other solutions come close to, but k8s shines at. You can go a long way without having reproducible multi-machine setups in different envs and can come pretty close when needed, with manual work.
> Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end.
If true, this is a strange design choice. I'd be wary of anything that was made complex just for the sake of it.
k8s gets enough flack without having to accuse it of being complex just for funsies.
> But it's still by far the best we've got.
k8s is the best we've got when we want k8s. The trick is to not want k8s for the sake of wanting k8s.
There are times when k8s provides tremendous value. Most companies who decide to use it do not have the problems that k8s promises to solve, and never will. Sadly, sometimes it's because they've spent their time and money on unnecessary complexity like k8s instead of building a product that delivers value.
Our team is happy to write raw YAML and use kustomize, because we prefer keeping the config plain and obvious, but we otherwise pretty much follow everything here.
LOL so, rather than linting YAML, bring in a whole programming language runtime plus third party library, adding yet another vendor lock, having to maintain versions, project compiling, moving away from K8S, adding mental overhead...
I have drastically reduced the amount of errors, mistakes, bugs, plain old wtf-induced hair pulling, by just mandating avoidance of YAML (and Helm) and using Jsonnet. Sure, there was some up-front work to write library code, but afterwards? I had people introduced to JSonnet with example deployment on one day, and shipping production-ready deployment for another app the next day.
Something we couldn't get with yaml.
I was talking to someone from a local startup a couple weeks ago who was trying to explain their devops stack. The number of different tools and platforms they were using was in the range of 50 different things, and they were asking for advice about how to integrate yet another thing to solve yet another self-inflicted problem.
It was as though they forgot what the goal was and started trying to collect as much experience with as many different tools as they could.
hopefully jsonnet or that apple thing will get more traction and popularlity.
Managing k8s, for me at least, is a lot easier than juggling multiple servers with potentially different hardware, software, or whatever else. It’s rare that businesses will have machines that are all identical. Trying to keep adding machines to a pool that you manage manually and keep them running can be very messy and get out of control if you’re not on top of it.
k8s can also get out of control though it’s also easier to reason about and understand in this context. Eg you have eight machines of varying specs but all they really have installed is what’s required to run k8s, so you haven’t got as much divergence there. You can then use k8s to schedule work across them or ask questions about the machines.
People make big claims but then it's not declarative enough to look up a resource or build a dependency tree and then your context deadline is exceeded.