Google has a tool by the same name (autopilot) internally that does close to the same thing. There was a paper published on it 10 months ago talked about here: https://news.ycombinator.com/item?id=22980467
Probably the scaling aspect of k8s autopilot is the only part that is similar. The rest looks similar to functionality provided by other tools internally.
I can't fight the feeling that this is all circling back to the application server stuff that was popular for a time. And, really, I can't tell when those went so wrong. :(
1. Someone has an idea. It's alright. Really good for their use case. Someone else hears about it, likes it, and adapts it to a similar use case. So and so forth until the idea has a large user base
2. Employees of large companies hear about the idea and implement it
3. Marketing gets a hold of the idea, gives it a flashy name, and uses it in promotions
4. A majority of the loudest voices in the industry get on board so everyone starts forcing this idea on every imaginable use case
5. A lot of people realize this is all too much extra work without much benefit so they start looking for more appropriate solutions
6. return to 1
It's happened with mainframes, object oriented programming, services, micro-services, web-all-the-things, blockchain, steaming data (kafka), sql, nosql, containers, agile, VMs, cloud, and many other things.
It's just technology people are the worst at getting caught up today's fad and I wish we could try to not do that as much.
Good news, you can invest in my new startup where we are solving just that issue with a decentralized social technology blockchain oracle that will make and enforce these decisions, reducing tech churn and enhancing developer productivity.
You have a lot of tech companies digging up gold, and a whole lot more selling shovels to the rest of the crowd.
Most of the software tech world is bullshit. Most best practices are bullshit. And, if I were feeling a bit conspiratorial, I'd say the large tech companies do this on purpose. They promote fads that overburden any smaller company with more modest budgets, thus keeping competitors at an arm's length. Resume-driven-development plays a role in this as well.
A lot of companies following the FAANG cargo cult are digging their own grave and don't even know it. They don't realize they don't have the manpower for microservices. Or the calendar time to make it work and still get product out the door before their competitor that doesn't even unit test eats their entire lunch.
There's hope that companies that have to be competitive from poorer countries will push correct technologies. I see it happening in UK, Brazil, Russia etc.
But it's no good to be big company when Whatsapp can be build by one person you missed to hire.
> Marketing gets a hold of the idea, gives it a flashy name, and uses it in promotions
After a half-decade in the industry, I am beginning to realize there is a lot of bad engineering hiding behind marketing and emoji. So many of these web tools have nightmarish interfaces and add complexity that, for most of the industry, is unnecessary. But their docs pages are full of rocket ships and confetti.... I swear, seeing a page with rocket ship emoji has become such a turnoff. Why do we need our engineering handed to us sprinkled with decorations like a cupcake?
And the imposter syndrome that the entire industry seems to share dictates that a large percentage of developers feel like they need to be using these tools as a badge to wear saying, "Yes, I am with it and hireable."
Meanwhile, the technology itself keeps churning because there is now a profession full of people believing that creating and open sourcing the Next Big Thing in ____ Technology is the best way to move their career forward. People keep reinventing wheels because everyone is focused on making a name for themselves with the new; no one notices the person doing mundane maintenance on Rails or whatever.
I think it's been this way since the 70s/80s to some extent, after having done some reading on the history of the profession. I think it's just scaled with the number of programmers and the Internet has applied its intensification effects.
From the TA: The maximum number of pods per node is 32, as opposed to 110 on standard GKE.
This is sad.. I don't understand why providers do it. It makes it very expensive to run small staging clusters. There is some reference on the current state here:
Comparable-ish with Fargate. You probably wouldn't be using this with an eye to save money unless you think (or have measured) that you'd spend more on operations, security or compliance otherwise.
I think the single biggest mistake people make with Kubernetes is implementing it too soon. Last company I worked for spent piles of time fighting K8s when a simple well implemented cluster would have done the job.
It makes a lot of sense to build portable infrastructure, but you can scale a long ways with much simpler technologies.
I would never voluntarily deploy anything using k8s or similar cloud stacks and services. But in practice the bulk of the complexity isn't in pushing and running the actual software, it's in provisioning attached resources (disks, databases), sharing certificates and keys, routing and load balancing, etc.
Way too complex for my tastes, but if you're convinced you need a highly automated control plane for those things then pushing .deb or .rpm packages doesn't even come close to solving that problem.
As usual it comes down to how you define the problem. You can't dissuade people from using k8s by comparing and contrasting k8s to alternatives; you've already ceded the debate over how to define the problem at that point. W'ever its relative merits, k8s is a reasonable approach to the problem of automating the control plane for "scaleable" services.
Debs move state (and computation) and provide a way to resolve dependencies. Done right they also provide rollback. Debs can moved a whole lot more than just the software.
Not saying it is the answer to everything, but it can do a whole lot more than software. Most of our problems are self made we can unmake them by changing the rules we operate under.
> As usual it comes down to how you define the problem.
Totally.
Redefine the problem until the solution is tractable and simple. K8s is the problem to a solution.
Some ETL logic packaged in Docker containers with a bit of simple scheduling and orchestration.
K8s sounds like a good idea on paper - you get reproducibility and resilence "for free" - but then I found out I have to effectively roll my own everything with k8s anyways and went with Jenkins instead.
> scaling stateless web app backend nodes written in scripting languages
K8s solves a far wider problem space. Need to run a data store? Use a StatefulSet and PersistentVolumes. Need an occasional task? Jobs and CronJobs. Need to know what's happening? Metrics and logs have APIs. Load balancing? Ingress? Firewalls? Security?
If someone knew nothing of operating systems other than a class on MINIX, I suspect running a massive datacenter would be easier with k8s than running a medium system of debian boxes.
I hear this refrain consistently on HN but of the options I've tried, GKE is the most pleasant for my single person app.
- App Engine has a bunch of weird limitations and slow deploy times. Qualitatively, it feels like the spotlight has moved on.
- Running my own compute instances felt like reinventing Kubernetes, especially once you roll your own deploy mechanism and throw load balancing in the mix. I also don't buy that it's easier.
- Cloud run is promising but for database heavy apps it's a non-starter.
- GKE was pretty smooth. It feels like it gets a lot more love than App Engine. The UI was functional with lots of depth. Once I push a docker image, GKE updates the nodes to serve the latest version. Load balancing was a matter of ~3 yaml files at 10 lines a pop.
You are mistaking my point here. I'm not suggesting you replace K8s with some other thing which does the same things k8s does, but that you build out a simpler/ easier to maintain solution until you need something like k8s. Many Many Many businesses can do just fine without the kind of complexity K8s (or App Engine or GKE) brings to the table.
There are a lot of businesses that will never have to deal with dynamic scaling, engineering autoscaling into those solutions is pointless. If you need that kind of scaling then K8r is fantastic. My point is a lot of people turn to K8s well before they need to or without understanding why they might need it.
My place has Kubernetes for when we are ready to scale. I am not sure it has gone above the default two pods. It did cause problems with moderately large uploads (5gb), as the memory gets chewed up far faster when using multiple small machines versus one big machine. I am not sure we will ever actually need this level of scaling.
Google did not say "Kubernetes is too complex" but rather, they are making this new tool - called Autopilot - that is an abstraction layer on top of Kubernetes for certain types of applications / companies.
This Autopilot system still uses Kubernetes AFAICT.
El Reg's snark is what makes it fun to speak with as they do cut to the heart of it.
GKE Autopilot is still fundamentally Kubernetes and supports the k8s APIs. GKE has simplified a lot of Kubernetes tasks that can be a pain, such as upgrades & scaling, but as many people said on other threads Kubernetes definitely is not for all workloads. For workloads that actually do need to scale then Autopilot removes a lot of the time consuming tasks required to get Kubernetes running and staying running. There are restrictions so check the docs on if it would work for how you use k8s.
Disclosure: I run product for GKE at Google so I'm definitely not a neutral voice on this...
I almost replied "who would look at Kubernetes and think it needs more layers?" but it sounds like Autopilot is a new cloud service - the customer (hopefully) doesn't interact with the underlying Kubernetes layer. I guess it kinda makes sense.
View from a Googler: Autopilot stitches together existing GKE functions and is just a cluster creation decision. The layer it adds is automating the different tools together and SRE support magic so that we're monitoring it now.
I love The Register for that, they just add hyperbole to something which would never come out from a company itself, and often they are right.
K8s is too complex, and I say it as k8s dev(ops) who dreams the next great thing will come soon which will save me from piles of YAMLs and will bring back programming fun ;) /s
K8s isn't too complex for what it does and can do.
it's just that a lot of people think they have to use k8s because it's trendy even it doesn't fit at all their needs and scale. Then they whine it's too complex.
No, it is not a typical Register headline, it is a half of a typical Register headline. However, HN has limits on headline lengths and it has been cut instead of being editorialized which afaik is another HN guidelines violation.
I'm looking for something like that, but I'm afraid of using any google developer services, such as GCP, for personal projects. What if I breach their TOS somehow and get banned, or what if I didn't breach their TOS and still get banned?
Can't afford to test my luck until I've finished migrating all my accounts off my gmail.
I'd also need a new credit card, I'm pretty sure they link accounts via shared CC. And even then, I'm pretty sure they still link accounts via other means. Did I link my gcp email in my android gmail app? Did I use the same IP address for both accounts? Or any of the thousand ways google has to know two accounts belong to the same person.
Honestly, creating a second google account might itself increase the chances of getting banned. Nobody knows with Google, and that's the problem.
I'm using my main google account on GCP and just yesterday started to take precautions about this.
I created a new google account with no relation to my main one and added it as an owner to my GCP project, so hopefully in the event of either account being banned I can still access everything.
Cloud run is great. I'm using an Nginx image to serve my static website. However, if I remember correctly, you can only respond to HTTP(S). So though it may be enough for most usecases, it is not essentially equal to running any container on the cloud.
You can open just 1 https port but you can map whatever port in your docker container to that. Some websocket implementations work with a second port and that just doesn't work. But you should probably split those services into two. But if you use something that can mix websockets and normal https traffic over 1 port, it works great.
Websockets now work on cloud run. We implemented that last month. Cloud run does have a few limitations: no service discovery built in, no docker compose support, limited set options for CPU/memory, no persistent disk, etc. But it's great for things like a simple Spring Boot server or any kind of stateless service. But you won't be running redis or a database there. It's just not designed to do that. It's also not great for running batch jobs; we tried and our jobs kept getting killed/throttled. Use a vm for that.
Luckily, there are lots of other Google services for that that you can plugin for that sort of stuff. Cloudrun is great if you are planning to use those things.
Kubernetes is what you use when you want to mix stateful and stateless stuff so you can avoid depending on those services. That makes sense if you need to support multiple clouds or on premise installations. But otherwise, it's a lot of extra complexity and devops even before you consider the overhead of managing the kubernetes cluster. There are lots of companies that talk themselves into needing this where the need is arguably a bit aspirational. I've been on more than one expensive project where we served absolutely no traffic at all with hundreds of dollars worth of kubernetes clusters idling for months on end that had no realistic hopes of ever getting more than a very modest amount of traffic even if everything worked out as they planned.
For my usage (lightweight game servers), I need WebSockets, which are still in beta and they currently force WS connections to close after 1 hour, which is a dealbreaker for me.
Dokku looks super cool but it's still a lot more involved than Heroku. Heroku doesn't make you think about provisioning servers, updating infrastructure, or manually setting up common integrations like backups and logging.
Just to illustrate the point, dokku's docs[0] for logging say "Warning: The default docker-local scheduler will "store" these until the next deploy or until the old containers are garbage collected - whichever runs first. If you require the logs beyond this point in time, please ship the logs to a centralized log server."
Alright, well, that's gonna be a whole lot more work than two click in Heroku to send all my logs to any logging provider of my choice.
In my day, we used to just insert the server into the rack, plug it into the network, terminal into it and configure it. I'm not being sarcastic here. There was a clear correspondence between what we were doing and what it meant for the infrastructure. Now there are so many layers of abstractions that we've basically forgotten it's all just CPUs, hard disks, memory modules and network connections.
Even if your default starting stack is somewhat complex, for example, separate client and server apps, database, cache, and queue, most one-click cloud providers (ex Heroku or ElasticBeanstalk) offer ways of unifying logs, monitoring, simple provisioning, etc. You are “locked-in” in that you can’t move to another provider within an hour, but the lock-in would still be very low, you can use generic technologies (ex: Memcached or Redis) and have an impact only on a few config files...
I’m not telling people what to do, if you like K8s or Docker or what have you knock yourself out, and I mean it. For example, people keep telling other people on HN not to use React and that’s a hill I’d die on - and could write a dissertation defending it. I’m just wondering what the dev experiences of others are so I may learn from them.
My background is in embedded, so I admittedly know extremely little about web development, but whenever I'm curious and sit down to read about microservices and containers and orchestration and all that stuff, my mind starts to numb and I can't help but conclude that 99% of companies that use it probably don't need to. And that they're just a complex way for engineers to keep themselves spinning their wheels and not actually working on an application. Like that guy who insists on doing nothing but refactoring and rearchitecting and moving things from one layer in the stack to the other, re-writing in different languages, integrating new third party libraries that do the same thing their existing third party libraries, but not actually adding anything that improves the product. How did the world get into this situation where you need all this complexity just to deploy and configure not even an application, but part of an application? Amazing!
Running a cluster across your IOT fleet (1k+ devices; 5-10 apps each) gives a nice interface for pushing out tasks, choosing applications for a device, configuring supervisor-device relationships, etc. It turns out from scratch Docker containers on ARM are very portable.
I think you’re being dramatic — embedded is notorious for crazy builds of weird config flags to even get “hello world” to compile, and you’re waving your arms pretending that concepts from Erlang abstracted away from the language are too much.
Docker is just cgroups and namespaces, with a zip file of code. More or less literally.
Orchestration is the same mess it’s always been — back to at least the telephone days, when Erlang used the same concepts.
> How did the world get into this situation where you need all this complexity just to deploy and configure not even an application, but part of an application? Amazing!
Because everyone else is doing it.
Things have gotten so complex that we need complex solutions, but w're afraid to make and own the solution ourselves because of cost, focus on core business or recruitment/knowledge concerns. So we need to find the next best open source solution to leverage the collaborative effort to reduce time and cost and have a large community to fall back on for support. And because everyone jumps on board of the same train all use cases must/will be accounted for (else the solution will fade into a niche), meaning the solutions becomes a problem in and of itself.
> And that they're just a complex way for engineers to keep themselves spinning their wheels and not actually working on an application...How did the world get into this situation where you need all this complexity
Isn't this just labeling the knowledge that you don't have (and could probably read up on) as potentially unnecessary complexity?
I mean, every time I hear about embedded, I keep hearing about byte boundaries, RTOSes, compiler chains, musl, JTAG, and a million other things that make my mind numb. But I assume embedded engineers need to know all those things, because of the constraints unique to their field. Some of them could be just "a complex way for engineers to keep themselves spinning their wheels and not actually working on an application", but a lot of them provide value in the real world and solve some specific problem.
Cloud infrastructure management frameworks are the same.
Fair enough--we perceive things we're ignorant about as complex. I buy that.
I guess I was comparing it to how things were (or more precisely, how I perceived things) 20 years ago. You'd rack a server or two, install Linux, stick the application in /usr/local/bin, make sure Apache was set up, and you were off and running. Simple enough. It probably didn't scale to 2020-sized Internet user counts, though.
> You'd rack a server or two, install Linux, stick the application in /usr/local/bin, make sure Apache was set up, and you were off and running. Simple enough.
That sounds incredibly complex! :) Where would you find a place to rack a server? Are there minimum rates for such a place, or can you rack a single server for one month? Would you be able to rack it yourself? What if there was some problem with the PSU, and the server lost power when the data center was fine? What if the network card failed? What about if a hard drive in the server blew out?
Cloud infra in 2021 infrastructure has pretty turnkey answers to all of those (Kubernetes is not one for any of the ones I mentioned, except maybe for “what happens when any node goes down“). So yes, the complexity might be high, but so are the capabilities.
I mean, maybe you are running a mom-and-pop shop with a server rack in the basement for some reason, and have pretty daytime-specific hours. That's fine, and I still think you can do the server rack thing. Or at least that's the way I see it.
Perhaps one piece of meta-commentary here is that server hardware has not become substantially cheaper, faster, smaller, and easier to set up since 20 years ago. I can't really set up a server in my small bedroom that's capable of serving 10k concurrent users, and doesn't deafen me with its noise. Maybe if I did, all this cloud stuff might be less ubiquitous?
Looking at the istio-linkerd-traefik-consul-whatever-the-heck mess of cloud-native ecosystem projects, I want to decree that backend engineers are no longer allowed to make fun of Javascript engineers for having too many frameworks anymore.
All the tools you mention are not integrated into your app neither change how you write code. In fact your application shouldn't even know what service mesh you are using.
I can write a back end application once and then have it run with istio/linkerd/traefik/whatever with zero code changes.
That doesn't happen in the Javascript world. The choice of framework directly affects your code.
> I just use and push code to Heroku and I'm done for the day, simple. NoOps I call it.
I just enter my Honda Civic and drive to my office. Simple. I call it "Sedan".
(Honda Civic driver looking at an 18-wheeler in the highway and not understanding why somebody would use that)
In all seriousness though, Kubernetes solves a specific set of problems. Just because you personally don't have these problems, doesn't mean that Kubernetes is bad, or that people who use it, don't need it.
Can you explain the whole concept of Kubernetes to someone whose knowledge of computers is limited to making simple webpages with HTML and using Excel/VBA?
In a 'normal' scenario, you might host your web page on an Nginx web server, on a Linux server somewhere. A container lets you do the same thing, but in an isolated area of the operating system. That means you can have an isolated area which has your web pages, the Nginx web server, and some other dependencies, all grouped up together into a distributable package called an image. And then you can pull that image on other servers and just start running it and it'll have your web pages and your Nginx running nicely.
So far that's Docker, or containers to put it more generally.
Now if your web page is so amazing that it receives a lot of traffic, your little container is going to get overwhelmed. And if it falls over, then it's dead and nobody can see your web page until you bring the container back up. Fortunately there are tools that let you manage this aspect of the image, called orchestration. You can tell orchestration tools how to figure out if an image is unhealthy and needs replacement, and if it falls over whether to bring it back, and importantly, how many copies of the image to run to handle the traffic. And if you need to push an updated image with an updated web page in it, how to gently make that new container available to the world without interrupting the traffic.
There's more to orchestration, you also tell containers how to talk to each other if needed, how to manage secrets, encryption, load balancing. There are lots of aspects of hosting that fall into this.
The two main orchestration tools I know of are Docker Swarm and Kubernetes. Docker Swarm is bundled with Docker already. It's pretty easy to shift from normal Docker use to Docker Swarm use, it works well enough for small-medium deployments. Kubernetes is a tool for much larger and highly flexible use cases, and it has a lot of levers and buttons and swiss army knives with its own swiss army knives. Many aspects of Kubernetes like the load balancing and secrets are all pluggable and you can use different tools in there.
Now you're at this article's topic, which is Kubernetes. K8s as it's called has a larger mindshare of the ops world, therefore everyone wants to use it, but it's very complicated, so a tool has been introduced to try to simplify it.
You know how there are linked references in Excel files that refer to other Excel files? What if you want to keep multiple copies of your Excel file with macros that refer to different files with data? And so you can run them on Windows, Mac or in your browser?
Kubernetes basically lets you define your references not as "c:\jon\reports\fy2020_final_final_2_comments_review_Bob_final.xlsx", but as "fy-report", with "fy-report" being defined elsewhere.
This is required to help you run the same program in different circumstances without breaking everything. You can say "run it on this slow computer with this test data", or you can "run it on many big computers with real data", but the program is exactly the same.
What makes it so complicated is that Kubernetes tries to abstract everything a given program would need, so it's not just references to external files, but practically the whole computer with all its network connections that must be defined elsewhere using Kubernetes' special language.
A lot of this special language is the same for 99% of programs, like in Excel you want your VLOOKUP to work on a $-pinned range with the last parameter set to FALSE 99% of the time. This makes people make the same stupid mistakes and finding them is hard.
And of course, this special language means you have to relearn a lot you know about running programs on computers, like when you move from Excel formulas to VBA.
Containers don’t have to be complex. Docker and docker-compose are very simple to use. Docker swarm (rip) and Nomad are similar to kubernetes but orders or magnitude simpler.
I used to be a Docker fanboy. But after writing a few semi-complex Dockerfiles, and seeing more of how complex and FUBAR things like networking is [1], I changed my tune.
This was years ago, so maybe they greatly simplified things. But somehow I doubt it =/
[1] " Docker, by default, punches massive holes through your firewall in non-obvious ways. People don't realize that with a default Docker configuration, containers are ignoring any normal firewall rules you may have setup with iptables or ufw." - https://news.ycombinator.com/item?id=25834444
I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot:
- I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store.
- I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I never clean those up and my Node simply fills up. Also I add some additional docker networks that break Pod to Pod networks.
- Istio is nice - why we don't add automatic injection for Pods in all namespaces? Including kube-system? And then they brick kube-proxy and the cluster stops working.
- I can use validating webhooks for better security? Lets watch on all resources. To keep it more secure lets set the failure policy of the webhook to Fail, so we never admit any modification without the apiserver to make a call to out webhook. Whats that? My single replica webhook has was evicted from the Pod (we didn't add any resource requests and limits) and now it cannot even be created or scheduled because kube-controller-manager and kube-scheduler cannot update their lease and they lost leadership and now are idling, effectively bricking the entire cluster.
Google would reduce the pain points with this change, however they would still face countless other issues with Kubernetes.
Oh yes secret management with kubectl is needlessly complicated
Sure just put your secret data on a file then we'll use your file name as the key of the secret.
Cronjobs sometimes have weird bugs as well.
A lot of its complexity is due to the fact it's an evolving system, that's fine. But I see that some things end up way more complex or unreliable than it needs due to overengineering or use cases no one needs
I fully agree with your points and would sum them up as "Kubernetes has a steep learning curve, a (quite) large interface and ample opportunities to shoot yourself into the foot with it" (plus, they're very funny).
However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of.
If you actually go all-in and fit your application to the principles of Kubernetes-native applications (instead of the other way around), then it works nothing short to amazing.
We're running 120 microservices in GKE and the difference to our custom-built setup before is night and day. I let my Infra team go surfing together for two weeks because without changes it flies mostly on autopilot.
Let's not kid ourselves, distributed computing is _hard_ and Kubernetes is a testament to that. I'm not saying it can't be made more accessible by further standardization, but there are fundamental limits to how easy it can be made.
Which by the way is leading to my only pet peeve with it: I feel most of the complexity of K8S comes from the fact that it got hyped as an enterprise product and then lots of features were built that support shoving your non cloud-native workload into Kubernetes even if it was never designed for it.
If you don't do or need all of that, the amount of interface, complexity and footguns shrinks significantly. Maybe it's time to better pull them apart in the documentation.
> If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of.
This argument basically sums up to "Developers just need discipline, and stop blaming the tools". While this is a sound argument on paper, the intrinsic complexity of software systems make it hard to pin the blame on developers. BTW This is the same argument Uncle Bob makes which is not so popular with many mainstream developers.
I get what you're saying, but my point is a bit more nuanced:
If your goal is to build highly reliable and available services to end users that are secure and scalable with a team of more than 10 engineers, eventually you will run into more than 50% of the concepts in Kubernetes anyway and end up re-inventing them.
Scaling up and down, node draining, finding out whether services are healthy, RBAC, resource distribution, secrets management, service hardening, introspection capabilities, explicit declaration of dependencies and endpoints and many, many more.
My point is: Sure, if your goal isn't that, it doesn't make sense to start out using Kubernetes.
But if at least eventually that's what you need, imho it's way preferable to just learn and apply well proven abstractions instead of reinventing the wheel along the way and end up with a less maintainable, capable and standardized solution you won't find anyone for maintaining.
If I hear about some of the comments here suggesting to "just spinning up docker-compose with Traefik in front" (disclaimer: I really like Traefik), then that reminds me of how some of the ops mess started that I historically had to care for.
Agreed, the truth usually lies somewhere in between and my point was we can't absolve the tools/ecosystems and put it on squarely on the devs. That definitely doesn't absolve teams and they need to do their homework before jumping on the bandwagon. K8s is great if you know what you're signing up for.
To a person who isn't using Erlang or ASP.net, suggesting that we should use either of those language packages and it will solve any of our problems without creating a thousand new ones sounds equally non-starterish to me.
To add a counter-example, I have lots of Ruby experience and I've just joined a Go team. I won't tell them to use Ruby, I will just do it where it makes sense and saves us time. (And then we'll have two problems... enter "limiting blast radius")
Point of my counter-example is, I'm extremely skeptical that all the world's problems can be solved by adopting a new mono-culture, whatever it is. There are 100% always gonna be some problems that are better solved in a different language. PHP is the best way to run Wordpress, for example (ok, so it's the only way to run Wordpress, but you get the idea... "Wordpress is the best way to..."), but I've been in high-functioning IT organizations that won't touch that with a ten foot pole, because "it's another language to support, and PHP is icky."
We also got rid of a perfectly fine Wiki in favor of centralized Knowledge-base software for similar reason. "Better to just have one KB. We don't need to be hosting another thing." So the chances of moving everything over to BEAM VM are next to nil, unless you are a product-focused company with just one product, or happen to have an absolute champion leading the effort to migrate all the things. For all the other things, you need to have a consistent answer too.
No tool is one-size-fits-all. Where Kubernetes shines most is under any environment that isn't running a single monolith or building a software monoculture and/or can't manage that for whatever reason (because those are all basic use cases that are frankly easy enough to manage without adding on top the additional complexity of Kubernetes; don't need it, don't use it!) IMHO, diversity in infrastructure is a plus though, and Kubernetes is a technology that it turns out enables this.
Ok, but we have a similar setup that runs on GCE instances. Deploying involves building an image and pushing a button. We don't really have the need for an Infrastructure team.
I once misconfigured iptables and locked myself out of our buildserver. Had to call lab support in a different country. Is Linux too complicated? Joyent famously took down their whole region by rebooting wrong nodes. It’s almost like running distributed networks of supercomputers at scale is hard or something...
I once did an `apt autoremove` on a custom install of CentOS handed to my team. Apt uninstalled python (and a lot more), and apt depends on python to run, so that was a bummer. The easiest way out was to reinstall the OS.
What makes you think Kubernetes "secrets" are appropriate for storing secret information? They're not secure (not without adding a bunch of other nonsense on top of them).
Because mutable persistence in Kubernetes can be super annoying to manage and people might grasp for whatever lifeline they can find.
If you have a managed object store or even relational database outside of k8s, the thought of storing arbitrary data in secrets probably doesn't come to mind. But if your enterprise spools up a cluster and tells you to use nfs PVCs with no other storage solution, suddenly you might start getting creative.
Kubernetes adds a vast amount of complexity, and in my rationale is because it centers scaling on the wrong unit (the Operating System).
Docker introduced a great level of abstraction and reproducibility over platforms. However, Docker (or OS-based containers) are the most atomic unit of computation on Kubernetes. Which causes centering scaling on the Instance, instead of the Application or even the functions.
This leads to a lot of unintended side-effects. Of which complexity is the most evident since now you need to handle scaling, monitoring and compliance over the VM layer rather than on the functions or the app.
I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit to allow proper scalability and a simpler and more powerful system on the long term.
With the exception that Wasm apps are based on a open standard, almost any language can target it, are more lightweight, and they can also run on the browser :)
I’m not sure I agree with the lightweight point. WebAssembly is missing features needed by many high-level languages. As a result, they have to resort to less than ideal tricks. For example, C# on WebAssembly runs using an interpreter even though normally it is JIT compiled.
To me, this essentially defines "the cloud" which exist because they've convinced CTOs that scaling is an infrastructure concern.
The scaling of their primary business (amazon.com, google.com) is at least partially an infrastructure problem - so they had to solve this anyways. Why not try to sell it too?
But it's very rare that you can scale a system by only scaling infrastructure. Ironically, probably the only way this will work is if you scale vertically - which means you don't need K8 and want to avoid the cloud like the plague.
There are all types of application-specific consistency issues that need to be treated as first class parts of the system for horizontal scaling to work.
> Which causes centering scaling on the Instance, instead of the Application or even the functions.
That should be why Red Hat took an early investment in Kubernetes. Operating systems did matter for them. Applications did not.
> I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit
I agree with that. The thing I worry about is a k8s variant like Krustlet, running Wasm instead of container, might introduce the same complexity into the Wasm world. We need a more app-centric scaling solution.
>I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit to allow proper scalability and a simpler and more powerful system on the long term.
Docker's co-founder agrees:
"If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing."
That's why I don't trust anything from google. They just release too complex software (cause theirs devs can deal with this crap anyway and it's good to let others suffer xD)
Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. I figure by the time any of my personal projects actually need to scale beyond 1 machine, I'd probably have enough revenue that I can afford to hire someone else to worry about it.
Maybe your starting instances are workhorses, but if you go from something like a t3.medium to two it’s a ~$60 / mth increase... Not something I’d personally optimize for.
Also why Docker in the first place? I’m genuinely wondering - in the stacks I run (Express / Python) it doesn’t seem necessary at low scale. Elastic Beanstalk, Heroku, Digital Ocean etc all offer facilities for single-command deploys that work out of the box.
I like Docker for the 'keeping it clean' aspect. Install php, composer and stuff juat because one of the projects you host requires it? Nope. Have to make excessive configuration on a system component just to run another application? Nope. Forgot how you set it up and now you are struggeling at the new machine? Use the Dockerfile.
GP was talking about projects that had revenue, and about "hiring someone" past a single instance.
I replied that beyond a single instance, you can probably get away with not hiring a K8s devops person and just spinning another instance. I'm not sure you've read this whole thing right.
And yes, I certainly wouldn't mind paying an additional $720 / yr for a project that had revenue; I almost certainly wouldn't want to spend money hiring a specialist, or spend time hyperoptimizing that myself - I make that in about a dozen hours of work, so counting how far one can go down the rabbit hole of optimizing server costs, and the associated cost of opportunity, the economics are crystal clear.
I don't have any successful personal projects but I have significant experience working with clients, and they are sold on the reasoning pretty much every single time ("I can charge you $3,000 for developing this feature, or we can use a paid service for $720 a year").
I also don't see how Docker is going to save you that much money; if you need a certain amount of compute, you need a certain amount of compute. AWS ElasticBeanstalk for instance charges nothing for spinning up an additional instance compared to EC2; there is no overhead for the PaaS aspect of it, like there would be in Heroku. Digital Ocean app platform is the same as EB, AFAIK.
GP talked about using Docker to do complex single host deployments until they needed horizontal scaling, which given max VM power, is after they can afford someone to manage it for them.
That makes me think they have multiple services of variable workload packed onto a single host, eg web server, async, and DB all on a single host via Docker.
That’s the antithesis of EB, which can only do horizontal scaling. Docker provides a way to replicate those multiple services in a deployment configuration when you want to set up a host image.
Not using Docker as an easy way to pack it all onto a single box as long as they can is just wasted expense.
I’d recommend giving Docker Swarm + Traefik a shot. It’s dead simple to set up manually and has very little “magic” in how things work under the hood. Plus much of your existing Docker Compose config will work out of the box. It vastly simplified the deployment process too.
I previously avoided Docker Swarm for ages since I assumed it involved the same level of complexity as k8s. I also initially figured that managed k8s would be a safer bet than managing my own Swarm cluster, but if you’ve used anybody’s managed k8s (or read https://k8s.af), you’ll realize that every cloud provider has their own closed source fork of k8s with plenty of nasty bugs that you can’t do anything about.
I use Swarm at home (only on a single node, because it turns out 3 are overkill for my needs) and it's been running great for 6 months so far. Before that I tried various incarnations of k8s and eventually they'd just destroy themselves up and require a rebuild (the main issue was persistent storage).
My only complaint with Swarm is there isn't an easy way to expose containers directly on the network (like host networking). I have a few containers (wireguard and minidlna) which need this, so those are running through docker-compose. I've tried macvlan but wasn't able to get that working in Swarm mode.
Maybe it has changed since I built it, but I wasn't able to get this working with Swarm services. I had to convert them to docker-compose to make it work. The docs suggest it should work with Swarm mode though, so maybe I need to try again.
Ideally I'd like to give each service its own IP on the network, which was possible with how I had k8s setup.
Asking because I never saw the point in multiple container replicas for simple self-hosted stuff. One container each has served me well so far [0] (Nextcloud, Bitwarden, GitLab), and if they crash, they just get restarted. Multiple containers increase throughput, supporting more users, is that it? It just sounds nightmarish in regards to storage and parallel, conflicting writes.
[0]: One container per component (web, db, cache, ...).
I do have just one container of each thing, but I was originally planning to have multiple nodes. I have 3x ThinkCenter Tiny M73 (with the Pentium CPU) and thought one would be a bit underpowered for everything I wanted to run (it was for k8s), so was planning to distribute services automatically across the swarm. One node is more than enough though, so I'd actually be fine with just docker-compose, but splitting everything up into separate 'services' is nice.
Docker Swarm is pretty much abandonware/on life support at best, so one should avoid using it for new stuff.
Hashicorp's Nomad is the best choice on the complexity for features scale IMHO, and that's why i'm writing an article how great it is, how easier some things are and what's missing compared to Kubernetes.
And yet, Docker Compose is pretty popular for local development, so much so, that it's not uncommon to find a docker-compose.yml in the repositories for many open source projects. And Docker Swarm builds on that, by bridging the gap between Docker Compose and multi-server deployments, with tools like Swarmpit and Podman for easier management of it as well, much like Rancher does for Kubernetes. I agree that Docker Swarm isn't developed as actively as it should be, but disagree that it should be avoided and disagree that it should be allowed to die. In my opinion, it's a more minimalistic and more sane approach to container orchestration with minimal up front investment (just install Docker and edit your Compose files a bit with deploy constraints, you're ready to go).
Hashicorp's Nomad is good if you have a strong engineering department or need to run mixed workloads (e.g. both containers and native processes) because its abstractions are well suited for this, but HCL, their DSL for describing deployments, doesn't map nicely to neither Docker, nor Docker Compose files, knowledgebases or tutorials. Nomad's integration with Consul is a major boon, but the need to run your own CA for safe communication between nodes, Nomad's read-only Web UI, and the oddity of HCL at times also makes it a non starter for me and some other people.
At the end of the day, these are just two data points, sadly the job market for Kubernetes also dwarfs everything else and sadly many companies will be burned by this and will learn nothing at the end of the day. Ideally, i think that the best route would be evaluating the orchestrators and other technologies that you want to use by doing pilot projects and such, and looking at them in real world circumstances, to determine their fit for your goals and needs (Web UIs will matter for some, but not for others, for example; as will onboarding and the need for long term investment vs plug and play).
Edit: as for Kubernetes, personally i find the K3s distribution to be an almost reasonable alternative to Swarm/Nomad, if the situation calls for it: https://k3s.io/
Edit #2: it would actually be pretty awesome to read more about your experience in this article that you're writing!
Absolutely. We use Swarm in production at ecoeats and it's a dream for simple clustering with multiple services. Using Hetzner clouds volume plugin gives EBS-like functionality too.
I quite like swarm for its compatibility with docker-compose, in particular to have the option of zero downtime deploys. If I wanted to manage a real cluster though I'd probably use nomad or GKE to avoid getting burned when the system is under load.
We use swarm for a small cluster in production a well. Extremely easy, zero downtime deployments are fantastic. I can explain it to someone else and quickly get them up to speed.
Having said that, the fact that it seems to be on life support has made me look at other alternatives, even a simple docker-compose per node.
Nomad's UI has a good number of functionalities that can be controlled through it. Sure, there are some more lower level operations that are CLI only(though this seems to be something they are actively working to improve on) but most of that probably won't be needed but someone just trying to run a couple containers on a single node.
Hmm, I guess I should shut down my business then since I'm a solo founder using k8s exclusively for my hosting for 3+ years...
Definitely a learning curve but honestly not bad if you like ops. Absolutely possible and beneficial for any size team. As always, it depends on what your goals are and how you want to use k8s.
In my opinion (and I'm biased as I work on GKE Autopilot), GKE is viable for a 1 developer project, especially with Autopilot mode (I have my own hobby projects deployed on GKE).
If you were self-hosting k8s, then I'd agree with you.
Long time ago I worked on HA setups for telecom (Wimax/LTE) equipment. Kubernetes is complicated but has nothing on those systems. Just to give you some idea - https://www.metaswitch.com/hs-fs/hubfs/Blogs/3gpp-ts-23-228-... (doesn’t even cover everything)
The very term "HA" still gives me nightmares. It can be very hard to get HA to work correctly. Many years ago, I worked in a startup and one of our main offerings was an HA network device. It was unbelievably finicky to get it to work in the first place and even harder to update the software on an HA cluster.
It's HA by intimidation. The cluster is complex enough that nobody even wants to touch it, and since human errors are the most common type of error out there, it breaks much less often.
Yes I believe this is why you see things like k3s in some iot/edge deployment scenarios. Because other alternatives for HA like OpenSAF have been severely lacking for years
It's the right attitude. The management fees alone for this autopilot thing are 0.10$ per hour. Or about 70$/month. It's a bargain considering all the hidden costs kubernetes imposes in terms of requiring people that know how to tame the complexity associated with it (i.e. very expensive devops people costing magnitudes more than that). Automating those people away is worth money.
I like Cloud Run for the same reason because I can use it without needing a lot of devops skills in my team or without sacrificing my own time (because I have those skills but have more valuable things to do). It allows me to focus on keeping my CI/CD pipeline (cloud run sets that up with a button click) busy with new functionality. And our hosting cost are close to 0$ because we stay below the freemium layer until we actually need to scale.
Hey, it's William from Google here. You're right about the costs, I just wanted to point out that Autopilot does include one cluster in GKE's free tier. So you'll only pay the ~$73/month if you have more than 1 cluster.
There's (almost) no limit to what you can run in one cluster too, and Kubernetes namespaces can help to separate different environments to allow for sharing.
Cloud Run sounds like the perfect solution for your workloads though!
docker swarm is a good alternative for you then, comes out of the box with docker and you only need to add a few lines to your docker compose files to make them swarm compatible
I’m leaning towards using Ubuntu/systemd to start all my services, and using a single container per project with SQLite.
This way:
- I can easily move from dev to prod by using the private container registry
- I have apt-get on the server and just use the default Ubuntu
- No distributed/network state
I used to use Core OS, but all these container OSes are here today and gone tomorrow, and normally have their own config standard. At least with Ubuntu they have LTS and a bunch of Google pages for fixing stuff.
systemd-nspawn is the most underrated piece of software on any modern link system. It just works and does exactly what it needs to do. Not to mention using a directory structure as a container image?! What is this sorcery?!
This is a honest question, but may I ask what you found to be hard about it?
For long I was scared of it because so many people say it's crazy complex. But actually it took no more than a dozen of hours for me to learn it and get a working setup on aws.
Maybe being full stack and having a strong knowledge of Linux and Docker helped.
Now I'm not pretending to be an expert with it, and there are certainly traps and mistakes that I didn't experience yet. But I don't understand what people find to be hard about it.
Lens has been a huge boon for helping us manage our kube cluster and regular devops operations, and even 15 minutes with it helped me grok a number of complex kubernetes concepts that I've struggled with for awhile now.
Everyone who works with k8s for a living should at least know of this tool imho its fantastic with prometheus
99.9 percent uptime for pods? That's 10 minutes downtime per week. We use simple VMs (on Google Cloud) at work and deploy services to them using Nix, and they have much less downtime than this. Does Kubernetes make it so difficult to do better than 99.9?
I can understand the use of kubernetes in very large orgs to manage clusters of hundreds of nodes, but it seems to me the complexity isn’t justified if you only have 1-100 say. There are lots of possibilities between 1 server and 100, and lots of ways to have simple replicable deploys if your needs are simple (probably 95% of businesses).
Simple load balancers without auto-scaling work fine!
For smaller non-critical services sometimes even one reliable server is an acceptable trade-off.
Do most companies really need kubernetes given the complexity and difficulty it introduces?
No, but it’s not just complexity. We’re a small company, but after spending about a month to setup our GKE a couple of years ago (incl learning terraform in the process), it’s been rock solid, low involvement and reliable. Declarative resource specification is a legitimate game changer and you couldn’t pay me to go back.
I'm not sure why you're getting downvoted unless people are afraid that you're sponsored by Google. Your experience aligns with mine. Running a SaaS app by myself that serves heavy loads around 1 rps with burst of 400+ rps, k8s works really great for my use case.
But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments. Plus, often times it's really unclear if I should scale vertically or horizontally.
"k8s works really great for my use case. [..] But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments."
TBO, it does not sound like it's working great for your use-case =) Wouldn't something like Cloud Run, AppEgine Standard or Heroku be much simpler and cheaper? Or just a single $5-15 per month* VM?
* depending on how much oomph you need for those 400 rps
Standardised declarative config/setup I've found really useful, declarative resource specification (as in n servers for service y) I don't really have a use for though I could see how in larger companies with hundreds of servers and lots of employees it would be attractive. If it is rock solid and reliable and impossible to mess up, I could see it being really nice (as your experiences sounds), though it feels to me like end users of services like GKE should not have to know about kubernetes or its complexity, nor have a chance to mess it up by editing configs directly.
I guess the ultimate goal of services like GKE autopilot is for you not to have to worry about kubernetes at all, just give them your workload and have it run on whatever resources they think is appropriate?
I do think it's important to recognise though that there are lots of ways to host services, and simpler with less abstractions is often better and more reliable and certainly easier to debug when things go wrong.
I mean yeah, I’m not arguing against autopilot and would probably have used it if it existed when we kicked off. Cloud Run had just come out and I really wanted to use that but the networking wouldn’t do what we needed, so had to go all in. But in the end, we learned enough Kube, we specified it all with Terraform, we haven’t tried to do too many fancy or complicated things, and we have reliable infrastructure fed with a simple deployment/build pipeline (just using cloud build, it’s clunky but it works). I mean maybe I’m conflicted as we’re part of GCS, but the system works well for us. If it didn’t we could’ve jumped to Azure or AWS and got credits there.
Maybe I can offer an answer to your question, I have worked at a couple of companies where we ran "small" scale k8s clusters (1-100 nodes as you say).
We have chosen k8s and I would again, because its nice to use. Its not necessarily easier, as you point out, the complexity of managing the cluster is considerable. But if you use a managed cluster like EKS or DO's k8s offering, you don't have to worry too much about the nodes and the unit of worry is the k8s config and then for deployment you can use Docker.
I like Docker, because its nice. Its nice to have the same setup locally as you have remotely.
In my experience the tooling around k8s is nice to manage declaratively, I never liked working with machines directly because even tools like Chef or Ansible feel very flimsy.
The other thing you can do is run on ECS or similar, but there the flexibility is a lot lower. So k8s for me offers the sweet spot of being able to do a lot quickly with a nice declarative interface.
I'd be interested to hear your take on how to best run a small cluster though.
Thanks, that's really interesting. Everyone has different challenges and requirements, and of course different experiences.
For smaller setups (say 1-10 services) I'm quite happy with cloud config and one VM per process behind one load balancer per service. It's simple to set up, scale and reproduce. This setup doesn't autoscale, but I've never really felt the need. We use Go and deploy one static binary per service at work with minimal dependencies so docker has never been very interesting. We could redeploy almost all the services we run within minutes if required with no data loss, so that bit feels similar to K8s I imagine.
For even smaller companies (many services at many companies) a single reliable server per service is often fine - it depends of course on things like uptime requirements for that service but not everything is of critical importance and sometimes uptime can be higher with a single untouched service.
I think what I'd worry about with a k8s config which affects live deployments is that I could make a tweak which seemed reasonable in isolation but broke things in inscrutable ways - many outages at big companies seem to be related to config changes nowadays.
With a simpler setup there is less chance of bringing everything down with a config change, because things are relatively static after deploy.
Sorry that should have said one binary per node really, not per service (though it is one binary per service, just on a few nodes for redundancy and load).
Services behind a load balancer so one node at a time replaced then restarted behind that, and/or you can do graceful restarts. There are a few ways.
They're run as systemd units and of course could restart for other reasons (OS Update, crash, OOM, hardware swapped out by host) - haven't noticed any problems related to that or deploys and I imagine the story is the same for other methods of running services (e.g. docker). As there is a load balancer individual nodes going down for a short time doesn't matter much.
> how do you deploy your static binary to the server? (without much downtime ?)
Ask yourself how would you solve this problem if you deployed by hand and automate that.
1. Create a brain-dead registry that gets information about what runs where (service name, ip address:port number, id, git commit, service state, last healthy_at). If you want to go crazy, do it 3x.
2. Have haproxy or nginx use the registry to build a communication map between services.
You are done.
For extra credit ( which is nearly cost free ) with 1. you now can build a brain-dead simple control plane by sticking an interface to 1 that lets someone/something toggle services automatically. For example, if you add percentage gauge to services, you can do hitless rolling deploys or cannery deploys.
I thought I read somewhere that the creator of WebSphere expressed similar opinions about his creation, and others have even called it mentally abusive.
Perhaps the enterprise class stuff is always just for large division of labor efforts. Like the "Liberty" stuff from WebSphere perhaps this really just saying if smaller groups and less people want to mess with it, it should have some new thought around more consolidated abstractions.
As Docker provided a simplified model with sensible defaults and a decent toolset for packaging and virtualizing single-node app environments into containers; so people went out and looked for a solution to virtualize multi-app workloads.
Honestly, I'm not sure Kubernetes is it. K8s seems like a leaky implementation detail when what people want is just to virtualize a workload (with similar simplicity to Docker) and have it work wherever the standard 'workload virtualiser' runs.
As someone who has never worked with containers, I find this rather amusing: Kubernetes is so complex that Google needs to roll out an "autopilot" feature, yet "it has won in the critically important container orchestration space". Makes you wonder what the alternatives must have looked like?!
Putting on my Asbestos Longjohns: The more I look into k8s ecosystem, the more I'm convinced that it's one of those things that suits FAANG etc, but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill. After all no one got fired for buying IBM and recommending Kubernetes. Most teams need a simpler deployment strategies that other's have succinctly mentioned elsewhere in this page.
I once have been told that development teams smaller than 20 developers have no business in using k8s, due to the complexity to brings. If something as essential as the infra so complex it is not readily understood by everyone on the team, a few (more than one) team members need to become the experts on the matter. For small teams this is simply not worth it.
Definitely; if you as a company want to do Kubernetes or even cloud services (beyond the easy managed service like Beanstalk or GCE), you need to have a dedicated expert on it. Or more abstractly, one full time unit (can be distributed). If it's some guy's part time hobby it will not work.
I think this is what went wrong with k8s. I saw lots of interests from hobbyist, and people proposing k8s in small teams. It become often hear that "you do containers in production? use k8s!" That's just a big disappointment waiting to happen.
As part of a small team currently using Kubernetes, I suspect it’s more how you use it - the tools and ecosystem have matured immensely in the last couple of years since I’ve first started using it.
I don’t think it suits all teams and use cases, but for us it’s absolutely fantastic and without going down the rabbit-hole of cloud-provider specific tools and recreating half the issues it solves, I’m not super sure what we’d use.
Agreed. I'm a solo technical founder and have been using k8s for all my hosting for 3+ years. It's so easy (for me) that I'm fine paying a premium for the managed service (GCP) since it saves me lots of time, my most valuable resource.
I've already climbed most of the learning curve so YMMV, but as a team of one and dozens of WordPress, MySQL, and bespoke app servers, kuberenetes makes ops manageable so I can spend time on things that really matter.
Deploying new web apps is trivial, declarative manifests are easy to reason about, TLS certs are issued and renewed automatically (cert-manager), backups are cheap and reliable (daily GCP snapshots), making changes to the cluster via declarative terraform is a breeze, etc etc. No way I could manage all the ops without leaning so heavily on the core foundation provided by k8s.
I’ve run a production kubernetes cluster that was hosting a DGraph cluster of 3 machines on its own, some ML workloads, and 4-5 products (each consisting of multiple services) and that was more than a single machine would have been able to handle.
Well _technically_, sure, we could have run a bunch of those products on a single machine, but there goes your durability and the memory overhead on some of them was quite Hugh, and properly fitting them onto a single machine would have required more optimisation and technical skills than the devs I was working with had or were inclined to do.
As in - a single workload that can't fit on a single physical machine? No I don't, although I certainly could if I needed to. Most of my workloads are either low-traffic WP sites or bespoke web-based business tools for clients with very bursty traffic.
Most of the value I get from k8s is the hands-off nature of it - I get slack notifications (prometheus+alertmanager) if anything is happening I need to address (e.g. workload down, node down, API not responding, etc). Otherwise I can safely ignore my cluster and know everything's good. Spinning up a new WP site takes 10m with backups, TLS, monitoring, etc built in.
If something as essential as the infrastructure is so complex that you need a dedicated expert on it, it's bad infrastructure. To take a offhand analogy, you don't need a dedicated highway maintainance engineer in order to drive your car.
> The more I look into k8s ecosystem, the more I'm convinced that it's one of those things that suits FAANG etc, but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill.
You are absolutely spot on because this is how not to pass the behavioral interview for Engineering Manager.
A friend of mine is a contributor to k8s itself, and of course, this all comes incredibly easy to them. Following their recommendation, I gave it a shot for my single-person, single-node (!) homelab, all without using MicroK8s, k3s or similar.
After a week of almost full-time work, I threw in the towel. Admittedly, I also had to learn concepts like reverse proxies alongside, too, so I was by no means well-equipped to begin with.
Yet, tossing together some docker-compose.yml files and "managing" them with a Python script has worked very well. Kubernetes really scarred me in that sense, and I am healed! Also, Caddy has helped me in actually enjoying configuring the webserver.
Yeah, of course running random docker compose files and containers from the internet and blissfully exposing your mongodb or whatnot service unsecured to the whole world seems like an easy, non-complicated alternative. Kubernetes has a few shitty defaults, like exposing a service account for all pods by default or allowing to mutate pod image tags, but most of the functionality it provides is a must have when you actually care about your SLA. Rolling updates with health check and configured back-off time? Separate ingress for OAM and live traffic with automatic HTTPS, etc? I could go on.
I am talking about a homelab, a single server at home, for home use. It's much safer now, with Docker compose, because I understand it and I wrote the core exposed part's configuration, the Caddyfile, myself, manually. I know exactly what's exposed, and it's exactly right the way it is!
The remaining risk comes from the services themselves having security holes, but k8s has that very same risk.
Ah ok. For a single node homelab setups I just throw everything on hostNetwork, second choice is NodePort (if there are port conflicts). In general k8s ingress on baremetal requires deeper understanding of its network design
I would (probably) spin up an ingress-controller on ports 80 and 443, using hostNetwork, then use Ingresses from then on (and as it's a single-node cluster, just create a wildcard DNS A record, and possibly an anchor for other CNAMEs to point at (depending on DNS server) pointing at the IP said ingress-controller is running on).
Does mean that anything that upsets the ingress controller is an outage, but for experimentation, that's probably OK.
but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill
There are very strong financial incentives for every individual developer and sysadmin to adopt Kubernetes, regardless of the impact it has on the organisation as a whole. In a sense this is engineering reaching the level of corporate maturity of the sales department who will optimise everything for their commission regardless of the organisations ability to deliver it at a profit, or even at all.
I'm sure there's a name to this phenomenon. Companies want stable software, and Regular Joe want better pay, but companies won't pay unless Joe starts doing crazy complex stuff that complicates things further.
Regular Joe learns complex stuff at your expense. He then leaves for greener pastures and higher pay thanks to the boost to his resume. You are then left with complex stuff you need to maintain and so you have to hire another Regular Joe for a higher salary than your first Regular Joe.
> There are very strong financial incentives for every individual developer and sysadmin to adopt Kubernetes, regardless of the impact it has on the organisation as a whole.
Then that organization is doing a terrible job of aligning incentives. I'm guessing their pay structure isn't terribly merit-based nor high enough that people aren't constantly thinking about other jobs.
If this is about FAANG (your comment wasn't, but others were), perhaps part of this is exposing larger problems in many smaller orgs. (note: I'm ex-FAANG and happily so)
Putting on my tinfoil hat: it suits FAANG to have potential competitors burn their runways on baroque tech fads like Kubernetes or <insert-react-state-management-architecture-of-the-month-here>. Extra credit if they end up hosting their overly complex solution on your platform.
I think Kubernetes in principle gets a lot of things very right - but it has over time grown into this huge amorphous blob of complexity that makes it very easy to shoot yourself in the foot with, as many people said :)
That issue is not endemic to Kubernetes, but rather to any larger system past a certain age, you learn stuff as you go along and would do stuff differently if you did it again today - but you can't easily, because you cannot break compatibility for everybody using your stuff.
As a concrete example from the Kubernetes world, there is a talk by Tim Hockin [1] about how today, they would fundamentally design the api-server differently and base pretty much everything on CRDs.
The industry and the k8s project are still figuring out the right way to do things that don't require the organization, size, and technical choices Google made.
From my experience it's actually sold as a simpler alternative to other infra provisioning. So you end up with situations where a team deploys whatever with a helm chart, and it sets up the stuff like magic and they build on it. Then when something goes wrong they literally have no idea how to fix anything and it becomes a waking nightmare.
I like Kubernetes, I don't overly like Helm charts because yes, they work, but you can install one without having to think about what's it putting in your cluster.
As a longtime and frequent user of k8s, I stay away from helm charts. I tried them out when they first got popular but I found they introduced more friction than they solved on the whole.
Not every addon/tool for the k8s ecosystem is worth it. I also don't bother with the ever-growing list of service meshes... not enough value to me for the overhead.
K8s is definitely the simpler alternative for me but there is still a lot of essential complexity in k8s due to the nature of the problems it's trying to solve. Mostly I like building on top of a solid foundation of standardized k8s API objects (pods, services, volumes, etc).
Tldr; Bring in only the add-ons and tools you really need so you don't add more complexity than necessary. Don't get swept up in the hype and marketing from other devs and cloud vendors.
This is such a great point and so frequently skimmed over in k8s discussions. We as tech folks tend to focus on the front page blog posts about 1000s of nodes and all of the orchestration that goes into complicated top 1% high-traffic/high-complexity use-case setups. In reality there are a lot of profitable businesses out there happily running a simple cluster set up with a single-digit number of deployments chugging along on it with zero down-time.
Really when looking at tools in the k8s ecosystem, it's better to approach it as you would importing a new library into your application. Most decent devs wouldn't blindly import a new lib so that they can copy/paste a single line of code they found online for a business critical function, and k8s tools should be no different. We must think about what value does a given tool bring, and is it worth the cost of learning/maintenance? Sometimes the answer is a resounding "yes", but too often the question isn't even asked.
k8s and its ecosystem represent a data center in software. Data centers are fairly complex constructs. It is then to be expected that this complexity will shine through in k8s' API, UI, UX. k8s' main mission seems to be to provide a complete digital data center, not an easy to use one, and I would argue that that is exactly the right choice. Over time, as the core of the beast is figured out, there will be (as there have been) more and more opportunities taken to actually help users navigate that complexity and/or resolve it into more natural and less error-prone interfaces. But in the meantime it seems like it's mostly on the community to provide (usually temporary) solutions for the most pressing usability concerns.
Kubernetes solves a very real and significant problem. But before you start using it, make sure you have the problem it solves.
If you're looking to have your small app eventually grow into a large one, read up on K8s and just make sure you're not blocking future-you from making your app work on it. E.g., work well in a container (which is useful for automated testing, deps management, etc), have a simple 'ping' endpoint to make sure the app is up, have a better config story than "recompile to change these variables", use a logging library, and tolerate any other services you're using to sometimes be down.
All useful things for a grown-up app to do anyways, all a bit of a PITA, and all better than trying to operate an app that doesn't do them.
Exactly this. Kubernetes is a service orchestrator, not a hosting platform. Those getting caught up in it just want a hosting platform, but those getting value out of it want a service orchestrator.
If you have one monolithic backend service (and most web applications really should start out this way), Kubernetes offers almost no benefits over alternatives.
In my opinion, k8s starts to shine when you have to manage hundreds of containers. When you have just dozens of them it's an overkill, but there's no way to smoothly slot in another solution between "docker-compose up -d" and spinning up a k8s cluster: you will (or think you will) hit a maintainability ceiling again and have to migrate to k8s.
There actually is, Hashicorp Nomad fits solidly in between those two options.
Nomad is way simpler to get a cluster up and running, has a great configuration syntax (I'll take HCL over YAML anyday) and had first class Terraform/Consul/Vault integrations.
Onboarding devs is fairly straightforward, if they can write a docker-compose.yml, it's an easy transition to a nomad job specification.
It took me by myself ~4 months to get our current hashistack(Vault/Consul/Nomad) stood up using Terraform+ansible. Two members of my team have been working to replace the hashistack with a self hosted K8's deployment and they just went over the 1 year mark and we still do not have something capable of hosting the workload currently running on the Hashistack.
This got a little long winded but I feel like this "it's docker compose or K8's, take your pick" mentality had led to a bunch of needless time being spent by smaller teams/companies on solutions that just aren't right for them.
Sorry to have to be the one to tell you: sometimes architectural decisions are driven by factors other than YAGNI. Right now you have throngs of young developers paying $50k+ a year for the privilege to learn how to use Docker and Kubernetes while in college, and when 90% of them inevitably get rejected from FAANG after graduation, you'll be able to hire them on the cheap and entice them with development stacks they're comfortable with.
I think that an important distinction is between deploying on k8s and operating it. For a small team (not measured in the dozens), the latter is unaffordable but the working style of the former is still powerful.
This feature helps a lot with that problem by bringing GCP closer to where AWS has been with Fargate. k8s will still be more work than using AWS ECS but it might also be preferable if you dislike using the provider’s components and want the control of, for example, doing your own load balancing and storage management.
What I think K8S (EKS, GKE, DO hosted environments at least) provide is a nice way to integrate things like Gitlab. This gives you a really easy to use CI/CD pipeline for very little work and configuration. This allows you to deploy production from your main branch and spin up feature branches that can be tested by the people that requested the feature very easily. This does not require an additional effort once the system is setup.
Also you can get red/green deployments and rolling deployments with little to no effort, which can be very nice, nice to have.
Complexity slows you down, regardless in which form it comes. You should only invest in a complex system/paradigm/language if you really need it. Before K8s OpenStack was all the rage, and I've seen several companies wasting millions trying to set up and operate their own cluster.
IMHO, most of what K8s offers can be obtained cheaper and in a simpler way using more "traditional" DevOps approaches and systems. I'm currently operating an IT infrastructure consisting of more than 20 different components, using only basic Linux technologies and open-source packages (ssh, iptables, ipsec, ferm, (r)syslog,...) plus Ansible to orchestrate it all. Never encountered a problem that I wasn't able to debug and fix within a few hours, and managed to have more than 99.99 % uptime so far. I understand that this approach might not work for large companies, but it seems to me a lot of startups are going down the K8s route just for the sake of it, and their DevOps processes become incredibly brittle and slow as a result.
Slightly off topic: I use Google Cloud’s built in auto-scaling for instance groups, and I’m very happy with it. It offers a reasonable GUI, and a way for me to scale VM instances up and down based on e.g. CPU usage. It seems to do 80% of what I want at 5% the complexity of Kubernetes.
What I miss the most is having my infrastructure defined as code, instead of via the GUI. But given that I have only four services (out of which two use preemptible VMs and only one needs to scale) it’s not really a problem — it wouldn’t take me many minutes to replicate this setup at another cloud provider.
You should try out Terraform—combined with managed instance groups it basically makes GCP into one big Kubernetes instance without having to manage it yourself.
448 comments
[ 4.0 ms ] story [ 143 ms ] threadDirect link to the docs: https://cloud.google.com/kubernetes-engine/docs/concepts/aut...
Google has a tool by the same name (autopilot) internally that does close to the same thing. There was a paper published on it 10 months ago talked about here: https://news.ycombinator.com/item?id=22980467
But likely they shared little in common.
1. Someone has an idea. It's alright. Really good for their use case. Someone else hears about it, likes it, and adapts it to a similar use case. So and so forth until the idea has a large user base
2. Employees of large companies hear about the idea and implement it
3. Marketing gets a hold of the idea, gives it a flashy name, and uses it in promotions
4. A majority of the loudest voices in the industry get on board so everyone starts forcing this idea on every imaginable use case
5. A lot of people realize this is all too much extra work without much benefit so they start looking for more appropriate solutions
6. return to 1
It's happened with mainframes, object oriented programming, services, micro-services, web-all-the-things, blockchain, steaming data (kafka), sql, nosql, containers, agile, VMs, cloud, and many other things.
It's just technology people are the worst at getting caught up today's fad and I wish we could try to not do that as much.
Most of the software tech world is bullshit. Most best practices are bullshit. And, if I were feeling a bit conspiratorial, I'd say the large tech companies do this on purpose. They promote fads that overburden any smaller company with more modest budgets, thus keeping competitors at an arm's length. Resume-driven-development plays a role in this as well.
A lot of companies following the FAANG cargo cult are digging their own grave and don't even know it. They don't realize they don't have the manpower for microservices. Or the calendar time to make it work and still get product out the door before their competitor that doesn't even unit test eats their entire lunch.
But it's no good to be big company when Whatsapp can be build by one person you missed to hire.
After a half-decade in the industry, I am beginning to realize there is a lot of bad engineering hiding behind marketing and emoji. So many of these web tools have nightmarish interfaces and add complexity that, for most of the industry, is unnecessary. But their docs pages are full of rocket ships and confetti.... I swear, seeing a page with rocket ship emoji has become such a turnoff. Why do we need our engineering handed to us sprinkled with decorations like a cupcake?
And the imposter syndrome that the entire industry seems to share dictates that a large percentage of developers feel like they need to be using these tools as a badge to wear saying, "Yes, I am with it and hireable."
Meanwhile, the technology itself keeps churning because there is now a profession full of people believing that creating and open sourcing the Next Big Thing in ____ Technology is the best way to move their career forward. People keep reinventing wheels because everyone is focused on making a name for themselves with the new; no one notices the person doing mundane maintenance on Rails or whatever.
I think it's been this way since the 70s/80s to some extent, after having done some reading on the history of the profession. I think it's just scaled with the number of programmers and the Internet has applied its intensification effects.
1. Build an opinionated solution that you control fully (e.g. difficult to fork).
2. Convince everybody to adopt it. Nobody gets fired for choosing $bigcorp
3. Grow it complex and expensive to maintain over time.
4. SELL software and services to manage its complexity.
...and the industry is getting more marketing/fad/resume-driven every day.
This is sad.. I don't understand why providers do it. It makes it very expensive to run small staging clusters. There is some reference on the current state here:
https://docs.google.com/spreadsheets/u/0/d/1yhkuBJBY2iO2Ax5F...
The pricing is... not cheap: https://cloud.google.com/kubernetes-engine/pricing
Comparable-ish with Fargate. You probably wouldn't be using this with an eye to save money unless you think (or have measured) that you'd spend more on operations, security or compliance otherwise.
It makes a lot of sense to build portable infrastructure, but you can scale a long ways with much simpler technologies.
Way too complex for my tastes, but if you're convinced you need a highly automated control plane for those things then pushing .deb or .rpm packages doesn't even come close to solving that problem.
As usual it comes down to how you define the problem. You can't dissuade people from using k8s by comparing and contrasting k8s to alternatives; you've already ceded the debate over how to define the problem at that point. W'ever its relative merits, k8s is a reasonable approach to the problem of automating the control plane for "scaleable" services.
Not saying it is the answer to everything, but it can do a whole lot more than software. Most of our problems are self made we can unmake them by changing the rules we operate under.
> As usual it comes down to how you define the problem.
Totally.
Redefine the problem until the solution is tractable and simple. K8s is the problem to a solution.
It's an insanely complex solution to a very niche problem - scaling stateless web app backend nodes written in scripting languages.
Stray even a little bit off the garden path and you start feeling pain.
K8s sounds like a good idea on paper - you get reproducibility and resilence "for free" - but then I found out I have to effectively roll my own everything with k8s anyways and went with Jenkins instead.
K8s solves a far wider problem space. Need to run a data store? Use a StatefulSet and PersistentVolumes. Need an occasional task? Jobs and CronJobs. Need to know what's happening? Metrics and logs have APIs. Load balancing? Ingress? Firewalls? Security?
If someone knew nothing of operating systems other than a class on MINIX, I suspect running a massive datacenter would be easier with k8s than running a medium system of debian boxes.
- App Engine has a bunch of weird limitations and slow deploy times. Qualitatively, it feels like the spotlight has moved on.
- Running my own compute instances felt like reinventing Kubernetes, especially once you roll your own deploy mechanism and throw load balancing in the mix. I also don't buy that it's easier.
- Cloud run is promising but for database heavy apps it's a non-starter.
- GKE was pretty smooth. It feels like it gets a lot more love than App Engine. The UI was functional with lots of depth. Once I push a docker image, GKE updates the nodes to serve the latest version. Load balancing was a matter of ~3 yaml files at 10 lines a pop.
There are a lot of businesses that will never have to deal with dynamic scaling, engineering autoscaling into those solutions is pointless. If you need that kind of scaling then K8r is fantastic. My point is a lot of people turn to K8s well before they need to or without understanding why they might need it.
Google did not say "Kubernetes is too complex" but rather, they are making this new tool - called Autopilot - that is an abstraction layer on top of Kubernetes for certain types of applications / companies.
This Autopilot system still uses Kubernetes AFAICT.
But it is a surprisingly negative headline. Autopilot sounds like a cool tool to simplify container orchestration!
That's the Register's schtick, they're snarky about everything.
GKE Autopilot is still fundamentally Kubernetes and supports the k8s APIs. GKE has simplified a lot of Kubernetes tasks that can be a pain, such as upgrades & scaling, but as many people said on other threads Kubernetes definitely is not for all workloads. For workloads that actually do need to scale then Autopilot removes a lot of the time consuming tasks required to get Kubernetes running and staying running. There are restrictions so check the docs on if it would work for how you use k8s.
Disclosure: I run product for GKE at Google so I'm definitely not a neutral voice on this...
Interested in having someone who has had to set up 1000s of containers in k8s and hated all of it?
K8s is too complex, and I say it as k8s dev(ops) who dreams the next great thing will come soon which will save me from piles of YAMLs and will bring back programming fun ;) /s
it's just that a lot of people think they have to use k8s because it's trendy even it doesn't fit at all their needs and scale. Then they whine it's too complex.
classic PEBCAK issue.
Are we just making rabbit holes out of rabbit holes of abstraction using kubernetes?
I just use and push code to Heroku and I'm done for the day, simple. NoOps I call it.
I wish more tools and platforms were like this.
https://cloud.google.com/run/
Can't afford to test my luck until I've finished migrating all my accounts off my gmail.
Honestly, creating a second google account might itself increase the chances of getting banned. Nobody knows with Google, and that's the problem.
I created a new google account with no relation to my main one and added it as an owner to my GCP project, so hopefully in the event of either account being banned I can still access everything.
https://cloud.google.com/run/docs/triggering/grpc
Luckily, there are lots of other Google services for that that you can plugin for that sort of stuff. Cloudrun is great if you are planning to use those things.
Kubernetes is what you use when you want to mix stateful and stateless stuff so you can avoid depending on those services. That makes sense if you need to support multiple clouds or on premise installations. But otherwise, it's a lot of extra complexity and devops even before you consider the overhead of managing the kubernetes cluster. There are lots of companies that talk themselves into needing this where the need is arguably a bit aspirational. I've been on more than one expensive project where we served absolutely no traffic at all with hundreds of dollars worth of kubernetes clusters idling for months on end that had no realistic hopes of ever getting more than a very modest amount of traffic even if everything worked out as they planned.
Just to illustrate the point, dokku's docs[0] for logging say "Warning: The default docker-local scheduler will "store" these until the next deploy or until the old containers are garbage collected - whichever runs first. If you require the logs beyond this point in time, please ship the logs to a centralized log server."
Alright, well, that's gonna be a whole lot more work than two click in Heroku to send all my logs to any logging provider of my choice.
[0] https://dokku.com/docs/deployment/logs/
https://azure.microsoft.com/en-us/services/app-service/conta...
(Not affiliated with either)
Edit: fixed name
When you start having a lot of pieces to manage eg. cache, database, auth, multiple applications then Kubernetes comes into its own.
Because then you can scale, monitor, trace, debug, log, backup, audit, encrypt and visualise all of those pieces in exactly the same way.
And do it irrespective of which cloud you use or whether it's even in the cloud at all.
I’m not telling people what to do, if you like K8s or Docker or what have you knock yourself out, and I mean it. For example, people keep telling other people on HN not to use React and that’s a hill I’d die on - and could write a dissertation defending it. I’m just wondering what the dev experiences of others are so I may learn from them.
Running a cluster across your IOT fleet (1k+ devices; 5-10 apps each) gives a nice interface for pushing out tasks, choosing applications for a device, configuring supervisor-device relationships, etc. It turns out from scratch Docker containers on ARM are very portable.
I think you’re being dramatic — embedded is notorious for crazy builds of weird config flags to even get “hello world” to compile, and you’re waving your arms pretending that concepts from Erlang abstracted away from the language are too much.
Docker is just cgroups and namespaces, with a zip file of code. More or less literally.
Orchestration is the same mess it’s always been — back to at least the telephone days, when Erlang used the same concepts.
Because everyone else is doing it.
Things have gotten so complex that we need complex solutions, but w're afraid to make and own the solution ourselves because of cost, focus on core business or recruitment/knowledge concerns. So we need to find the next best open source solution to leverage the collaborative effort to reduce time and cost and have a large community to fall back on for support. And because everyone jumps on board of the same train all use cases must/will be accounted for (else the solution will fade into a niche), meaning the solutions becomes a problem in and of itself.
Repeating the cycle once again.
Isn't this just labeling the knowledge that you don't have (and could probably read up on) as potentially unnecessary complexity?
I mean, every time I hear about embedded, I keep hearing about byte boundaries, RTOSes, compiler chains, musl, JTAG, and a million other things that make my mind numb. But I assume embedded engineers need to know all those things, because of the constraints unique to their field. Some of them could be just "a complex way for engineers to keep themselves spinning their wheels and not actually working on an application", but a lot of them provide value in the real world and solve some specific problem.
Cloud infrastructure management frameworks are the same.
I guess I was comparing it to how things were (or more precisely, how I perceived things) 20 years ago. You'd rack a server or two, install Linux, stick the application in /usr/local/bin, make sure Apache was set up, and you were off and running. Simple enough. It probably didn't scale to 2020-sized Internet user counts, though.
That sounds incredibly complex! :) Where would you find a place to rack a server? Are there minimum rates for such a place, or can you rack a single server for one month? Would you be able to rack it yourself? What if there was some problem with the PSU, and the server lost power when the data center was fine? What if the network card failed? What about if a hard drive in the server blew out?
Cloud infra in 2021 infrastructure has pretty turnkey answers to all of those (Kubernetes is not one for any of the ones I mentioned, except maybe for “what happens when any node goes down“). So yes, the complexity might be high, but so are the capabilities.
I mean, maybe you are running a mom-and-pop shop with a server rack in the basement for some reason, and have pretty daytime-specific hours. That's fine, and I still think you can do the server rack thing. Or at least that's the way I see it.
Perhaps one piece of meta-commentary here is that server hardware has not become substantially cheaper, faster, smaller, and easier to set up since 20 years ago. I can't really set up a server in my small bedroom that's capable of serving 10k concurrent users, and doesn't deafen me with its noise. Maybe if I did, all this cloud stuff might be less ubiquitous?
All the tools you mention are not integrated into your app neither change how you write code. In fact your application shouldn't even know what service mesh you are using.
I can write a back end application once and then have it run with istio/linkerd/traefik/whatever with zero code changes.
That doesn't happen in the Javascript world. The choice of framework directly affects your code.
I just enter my Honda Civic and drive to my office. Simple. I call it "Sedan".
(Honda Civic driver looking at an 18-wheeler in the highway and not understanding why somebody would use that)
In all seriousness though, Kubernetes solves a specific set of problems. Just because you personally don't have these problems, doesn't mean that Kubernetes is bad, or that people who use it, don't need it.
Containers add vast complexity, add another layer of complexity on top.
So far that's Docker, or containers to put it more generally.
Now if your web page is so amazing that it receives a lot of traffic, your little container is going to get overwhelmed. And if it falls over, then it's dead and nobody can see your web page until you bring the container back up. Fortunately there are tools that let you manage this aspect of the image, called orchestration. You can tell orchestration tools how to figure out if an image is unhealthy and needs replacement, and if it falls over whether to bring it back, and importantly, how many copies of the image to run to handle the traffic. And if you need to push an updated image with an updated web page in it, how to gently make that new container available to the world without interrupting the traffic.
There's more to orchestration, you also tell containers how to talk to each other if needed, how to manage secrets, encryption, load balancing. There are lots of aspects of hosting that fall into this.
The two main orchestration tools I know of are Docker Swarm and Kubernetes. Docker Swarm is bundled with Docker already. It's pretty easy to shift from normal Docker use to Docker Swarm use, it works well enough for small-medium deployments. Kubernetes is a tool for much larger and highly flexible use cases, and it has a lot of levers and buttons and swiss army knives with its own swiss army knives. Many aspects of Kubernetes like the load balancing and secrets are all pluggable and you can use different tools in there.
Now you're at this article's topic, which is Kubernetes. K8s as it's called has a larger mindshare of the ops world, therefore everyone wants to use it, but it's very complicated, so a tool has been introduced to try to simplify it.
Kubernetes basically lets you define your references not as "c:\jon\reports\fy2020_final_final_2_comments_review_Bob_final.xlsx", but as "fy-report", with "fy-report" being defined elsewhere.
This is required to help you run the same program in different circumstances without breaking everything. You can say "run it on this slow computer with this test data", or you can "run it on many big computers with real data", but the program is exactly the same.
What makes it so complicated is that Kubernetes tries to abstract everything a given program would need, so it's not just references to external files, but practically the whole computer with all its network connections that must be defined elsewhere using Kubernetes' special language.
A lot of this special language is the same for 99% of programs, like in Excel you want your VLOOKUP to work on a $-pinned range with the last parameter set to FALSE 99% of the time. This makes people make the same stupid mistakes and finding them is hard.
And of course, this special language means you have to relearn a lot you know about running programs on computers, like when you move from Excel formulas to VBA.
This was years ago, so maybe they greatly simplified things. But somehow I doubt it =/
[1] " Docker, by default, punches massive holes through your firewall in non-obvious ways. People don't realize that with a default Docker configuration, containers are ignoring any normal firewall rules you may have setup with iptables or ufw." - https://news.ycombinator.com/item?id=25834444
- I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store.
- I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I never clean those up and my Node simply fills up. Also I add some additional docker networks that break Pod to Pod networks.
- Istio is nice - why we don't add automatic injection for Pods in all namespaces? Including kube-system? And then they brick kube-proxy and the cluster stops working.
- I can use validating webhooks for better security? Lets watch on all resources. To keep it more secure lets set the failure policy of the webhook to Fail, so we never admit any modification without the apiserver to make a call to out webhook. Whats that? My single replica webhook has was evicted from the Pod (we didn't add any resource requests and limits) and now it cannot even be created or scheduled because kube-controller-manager and kube-scheduler cannot update their lease and they lost leadership and now are idling, effectively bricking the entire cluster.
Google would reduce the pain points with this change, however they would still face countless other issues with Kubernetes.
Sure just put your secret data on a file then we'll use your file name as the key of the secret.
Cronjobs sometimes have weird bugs as well.
A lot of its complexity is due to the fact it's an evolving system, that's fine. But I see that some things end up way more complex or unreliable than it needs due to overengineering or use cases no one needs
However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of.
If you actually go all-in and fit your application to the principles of Kubernetes-native applications (instead of the other way around), then it works nothing short to amazing.
We're running 120 microservices in GKE and the difference to our custom-built setup before is night and day. I let my Infra team go surfing together for two weeks because without changes it flies mostly on autopilot.
Let's not kid ourselves, distributed computing is _hard_ and Kubernetes is a testament to that. I'm not saying it can't be made more accessible by further standardization, but there are fundamental limits to how easy it can be made.
Which by the way is leading to my only pet peeve with it: I feel most of the complexity of K8S comes from the fact that it got hyped as an enterprise product and then lots of features were built that support shoving your non cloud-native workload into Kubernetes even if it was never designed for it.
If you don't do or need all of that, the amount of interface, complexity and footguns shrinks significantly. Maybe it's time to better pull them apart in the documentation.
This argument basically sums up to "Developers just need discipline, and stop blaming the tools". While this is a sound argument on paper, the intrinsic complexity of software systems make it hard to pin the blame on developers. BTW This is the same argument Uncle Bob makes which is not so popular with many mainstream developers.
You're right about feature creep in k8s though.
If your goal is to build highly reliable and available services to end users that are secure and scalable with a team of more than 10 engineers, eventually you will run into more than 50% of the concepts in Kubernetes anyway and end up re-inventing them.
Scaling up and down, node draining, finding out whether services are healthy, RBAC, resource distribution, secrets management, service hardening, introspection capabilities, explicit declaration of dependencies and endpoints and many, many more.
My point is: Sure, if your goal isn't that, it doesn't make sense to start out using Kubernetes.
But if at least eventually that's what you need, imho it's way preferable to just learn and apply well proven abstractions instead of reinventing the wheel along the way and end up with a less maintainable, capable and standardized solution you won't find anyone for maintaining.
If I hear about some of the comments here suggesting to "just spinning up docker-compose with Traefik in front" (disclaimer: I really like Traefik), then that reminds me of how some of the ops mess started that I historically had to care for.
It’s always easier to shift the blame somewhere else.
Thats why some radical but correct concepts are so hard to push.
To add a counter-example, I have lots of Ruby experience and I've just joined a Go team. I won't tell them to use Ruby, I will just do it where it makes sense and saves us time. (And then we'll have two problems... enter "limiting blast radius")
Point of my counter-example is, I'm extremely skeptical that all the world's problems can be solved by adopting a new mono-culture, whatever it is. There are 100% always gonna be some problems that are better solved in a different language. PHP is the best way to run Wordpress, for example (ok, so it's the only way to run Wordpress, but you get the idea... "Wordpress is the best way to..."), but I've been in high-functioning IT organizations that won't touch that with a ten foot pole, because "it's another language to support, and PHP is icky."
We also got rid of a perfectly fine Wiki in favor of centralized Knowledge-base software for similar reason. "Better to just have one KB. We don't need to be hosting another thing." So the chances of moving everything over to BEAM VM are next to nil, unless you are a product-focused company with just one product, or happen to have an absolute champion leading the effort to migrate all the things. For all the other things, you need to have a consistent answer too.
No tool is one-size-fits-all. Where Kubernetes shines most is under any environment that isn't running a single monolith or building a software monoculture and/or can't manage that for whatever reason (because those are all basic use cases that are frankly easy enough to manage without adding on top the additional complexity of Kubernetes; don't need it, don't use it!) IMHO, diversity in infrastructure is a plus though, and Kubernetes is a technology that it turns out enables this.
For example you still need a way to get BEAM onto hosts, still need to manage the OS on the host, still need to setup networking, RBAC etc.
In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident:
GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind
https://www.youtube.com/watch?v=30jNsCVLpAE
Why would someone want to store non-secret information as a secret?
— Douglas Adams
Top reason given to me by developers: "I don't want to spend time thinking about the distinction."
If you have a managed object store or even relational database outside of k8s, the thought of storing arbitrary data in secrets probably doesn't come to mind. But if your enterprise spools up a cluster and tells you to use nfs PVCs with no other storage solution, suddenly you might start getting creative.
One of the goal of K8S is normalization/standardization of a complex topic to better share knowledge
Just in case you haven't figured out the proper way to do this, you should use docker:dind.
Docker introduced a great level of abstraction and reproducibility over platforms. However, Docker (or OS-based containers) are the most atomic unit of computation on Kubernetes. Which causes centering scaling on the Instance, instead of the Application or even the functions.
This leads to a lot of unintended side-effects. Of which complexity is the most evident since now you need to handle scaling, monitoring and compliance over the VM layer rather than on the functions or the app.
I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit to allow proper scalability and a simpler and more powerful system on the long term.
Agreed, although at some point in a not very far feature most of those missing features will resolved. So in my mind is just a matter of time.
The Wasm Community group is doing an awesome work on that :)
The scaling of their primary business (amazon.com, google.com) is at least partially an infrastructure problem - so they had to solve this anyways. Why not try to sell it too?
But it's very rare that you can scale a system by only scaling infrastructure. Ironically, probably the only way this will work is if you scale vertically - which means you don't need K8 and want to avoid the cloud like the plague.
There are all types of application-specific consistency issues that need to be treated as first class parts of the system for horizontal scaling to work.
That should be why Red Hat took an early investment in Kubernetes. Operating systems did matter for them. Applications did not.
> I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit
I agree with that. The thing I worry about is a k8s variant like Krustlet, running Wasm instead of container, might introduce the same complexity into the Wasm world. We need a more app-centric scaling solution.
Docker's co-founder agrees:
"If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing."
https://twitter.com/solomonstre/status/1111004913222324225
Only thing that's somewhat good is Go lang.
Also why Docker in the first place? I’m genuinely wondering - in the stacks I run (Express / Python) it doesn’t seem necessary at low scale. Elastic Beanstalk, Heroku, Digital Ocean etc all offer facilities for single-command deploys that work out of the box.
Docker make it easy to run the same version of code in different places and let’s things run next to each other without version conflicts.
Also, I think you’re in a very small minority not to care about $720/yr increases in your hobbies.
I replied that beyond a single instance, you can probably get away with not hiring a K8s devops person and just spinning another instance. I'm not sure you've read this whole thing right.
And yes, I certainly wouldn't mind paying an additional $720 / yr for a project that had revenue; I almost certainly wouldn't want to spend money hiring a specialist, or spend time hyperoptimizing that myself - I make that in about a dozen hours of work, so counting how far one can go down the rabbit hole of optimizing server costs, and the associated cost of opportunity, the economics are crystal clear.
I don't have any successful personal projects but I have significant experience working with clients, and they are sold on the reasoning pretty much every single time ("I can charge you $3,000 for developing this feature, or we can use a paid service for $720 a year").
I also don't see how Docker is going to save you that much money; if you need a certain amount of compute, you need a certain amount of compute. AWS ElasticBeanstalk for instance charges nothing for spinning up an additional instance compared to EC2; there is no overhead for the PaaS aspect of it, like there would be in Heroku. Digital Ocean app platform is the same as EB, AFAIK.
That makes me think they have multiple services of variable workload packed onto a single host, eg web server, async, and DB all on a single host via Docker.
That’s the antithesis of EB, which can only do horizontal scaling. Docker provides a way to replicate those multiple services in a deployment configuration when you want to set up a host image.
Not using Docker as an easy way to pack it all onto a single box as long as they can is just wasted expense.
I previously avoided Docker Swarm for ages since I assumed it involved the same level of complexity as k8s. I also initially figured that managed k8s would be a safer bet than managing my own Swarm cluster, but if you’ve used anybody’s managed k8s (or read https://k8s.af), you’ll realize that every cloud provider has their own closed source fork of k8s with plenty of nasty bugs that you can’t do anything about.
My only complaint with Swarm is there isn't an easy way to expose containers directly on the network (like host networking). I have a few containers (wireguard and minidlna) which need this, so those are running through docker-compose. I've tried macvlan but wasn't able to get that working in Swarm mode.
Ideally I'd like to give each service its own IP on the network, which was possible with how I had k8s setup.
Asking because I never saw the point in multiple container replicas for simple self-hosted stuff. One container each has served me well so far [0] (Nextcloud, Bitwarden, GitLab), and if they crash, they just get restarted. Multiple containers increase throughput, supporting more users, is that it? It just sounds nightmarish in regards to storage and parallel, conflicting writes.
[0]: One container per component (web, db, cache, ...).
Hashicorp's Nomad is the best choice on the complexity for features scale IMHO, and that's why i'm writing an article how great it is, how easier some things are and what's missing compared to Kubernetes.
Hashicorp's Nomad is good if you have a strong engineering department or need to run mixed workloads (e.g. both containers and native processes) because its abstractions are well suited for this, but HCL, their DSL for describing deployments, doesn't map nicely to neither Docker, nor Docker Compose files, knowledgebases or tutorials. Nomad's integration with Consul is a major boon, but the need to run your own CA for safe communication between nodes, Nomad's read-only Web UI, and the oddity of HCL at times also makes it a non starter for me and some other people.
At the end of the day, these are just two data points, sadly the job market for Kubernetes also dwarfs everything else and sadly many companies will be burned by this and will learn nothing at the end of the day. Ideally, i think that the best route would be evaluating the orchestrators and other technologies that you want to use by doing pilot projects and such, and looking at them in real world circumstances, to determine their fit for your goals and needs (Web UIs will matter for some, but not for others, for example; as will onboarding and the need for long term investment vs plug and play).
Edit: as for Kubernetes, personally i find the K3s distribution to be an almost reasonable alternative to Swarm/Nomad, if the situation calls for it: https://k3s.io/
Edit #2: it would actually be pretty awesome to read more about your experience in this article that you're writing!
> Swarmpit and Podman
I actually meant Swarmpit ( https://swarmpit.io/ ) and Portainer ( https://www.portainer.io/ ).
Podman is another container runtime that acts as an alternative to Docker (even if it is not feature complete), so i misspoke.
Nomad's UI has a good number of functionalities that can be controlled through it. Sure, there are some more lower level operations that are CLI only(though this seems to be something they are actively working to improve on) but most of that probably won't be needed but someone just trying to run a couple containers on a single node.
In places with 10+ developers and 10+ services on a single cluster it works surprisingly well from the user (developer) side.
Definitely a learning curve but honestly not bad if you like ops. Absolutely possible and beneficial for any size team. As always, it depends on what your goals are and how you want to use k8s.
If you were self-hosting k8s, then I'd agree with you.
I like Cloud Run for the same reason because I can use it without needing a lot of devops skills in my team or without sacrificing my own time (because I have those skills but have more valuable things to do). It allows me to focus on keeping my CI/CD pipeline (cloud run sets that up with a button click) busy with new functionality. And our hosting cost are close to 0$ because we stay below the freemium layer until we actually need to scale.
Edit. corrected the typo 700->70
for these case, i still have gke cluster around.
(Still agree that Cloud Run isn't for everyone.)
There's (almost) no limit to what you can run in one cluster too, and Kubernetes namespaces can help to separate different environments to allow for sharing.
Cloud Run sounds like the perfect solution for your workloads though!
This way:
- I can easily move from dev to prod by using the private container registry
- I have apt-get on the server and just use the default Ubuntu
- No distributed/network state
I used to use Core OS, but all these container OSes are here today and gone tomorrow, and normally have their own config standard. At least with Ubuntu they have LTS and a bunch of Google pages for fixing stuff.
For long I was scared of it because so many people say it's crazy complex. But actually it took no more than a dozen of hours for me to learn it and get a working setup on aws.
Maybe being full stack and having a strong knowledge of Linux and Docker helped.
Now I'm not pretending to be an expert with it, and there are certainly traps and mistakes that I didn't experience yet. But I don't understand what people find to be hard about it.
So, not simple.
Lens has been a huge boon for helping us manage our kube cluster and regular devops operations, and even 15 minutes with it helped me grok a number of complex kubernetes concepts that I've struggled with for awhile now.
Everyone who works with k8s for a living should at least know of this tool imho its fantastic with prometheus
Simple load balancers without auto-scaling work fine!
For smaller non-critical services sometimes even one reliable server is an acceptable trade-off.
Do most companies really need kubernetes given the complexity and difficulty it introduces?
But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments. Plus, often times it's really unclear if I should scale vertically or horizontally.
TBO, it does not sound like it's working great for your use-case =) Wouldn't something like Cloud Run, AppEgine Standard or Heroku be much simpler and cheaper? Or just a single $5-15 per month* VM?
* depending on how much oomph you need for those 400 rps
I guess the ultimate goal of services like GKE autopilot is for you not to have to worry about kubernetes at all, just give them your workload and have it run on whatever resources they think is appropriate?
I do think it's important to recognise though that there are lots of ways to host services, and simpler with less abstractions is often better and more reliable and certainly easier to debug when things go wrong.
If I hear more reports like this I might just have to try out GKE :)
We have chosen k8s and I would again, because its nice to use. Its not necessarily easier, as you point out, the complexity of managing the cluster is considerable. But if you use a managed cluster like EKS or DO's k8s offering, you don't have to worry too much about the nodes and the unit of worry is the k8s config and then for deployment you can use Docker.
I like Docker, because its nice. Its nice to have the same setup locally as you have remotely.
In my experience the tooling around k8s is nice to manage declaratively, I never liked working with machines directly because even tools like Chef or Ansible feel very flimsy.
The other thing you can do is run on ECS or similar, but there the flexibility is a lot lower. So k8s for me offers the sweet spot of being able to do a lot quickly with a nice declarative interface.
I'd be interested to hear your take on how to best run a small cluster though.
For smaller setups (say 1-10 services) I'm quite happy with cloud config and one VM per process behind one load balancer per service. It's simple to set up, scale and reproduce. This setup doesn't autoscale, but I've never really felt the need. We use Go and deploy one static binary per service at work with minimal dependencies so docker has never been very interesting. We could redeploy almost all the services we run within minutes if required with no data loss, so that bit feels similar to K8s I imagine.
For even smaller companies (many services at many companies) a single reliable server per service is often fine - it depends of course on things like uptime requirements for that service but not everything is of critical importance and sometimes uptime can be higher with a single untouched service.
I think what I'd worry about with a k8s config which affects live deployments is that I could make a tweak which seemed reasonable in isolation but broke things in inscrutable ways - many outages at big companies seem to be related to config changes nowadays.
With a simpler setup there is less chance of bringing everything down with a config change, because things are relatively static after deploy.
how do you deploy your static binary to the server? (without much downtime ?)
Services behind a load balancer so one node at a time replaced then restarted behind that, and/or you can do graceful restarts. There are a few ways.
They're run as systemd units and of course could restart for other reasons (OS Update, crash, OOM, hardware swapped out by host) - haven't noticed any problems related to that or deploys and I imagine the story is the same for other methods of running services (e.g. docker). As there is a load balancer individual nodes going down for a short time doesn't matter much.
Ask yourself how would you solve this problem if you deployed by hand and automate that.
1. Create a brain-dead registry that gets information about what runs where (service name, ip address:port number, id, git commit, service state, last healthy_at). If you want to go crazy, do it 3x.
2. Have haproxy or nginx use the registry to build a communication map between services.
You are done.
For extra credit ( which is nearly cost free ) with 1. you now can build a brain-dead simple control plane by sticking an interface to 1 that lets someone/something toggle services automatically. For example, if you add percentage gauge to services, you can do hitless rolling deploys or cannery deploys.
Perhaps the enterprise class stuff is always just for large division of labor efforts. Like the "Liberty" stuff from WebSphere perhaps this really just saying if smaller groups and less people want to mess with it, it should have some new thought around more consolidated abstractions.
Honestly, I'm not sure Kubernetes is it. K8s seems like a leaky implementation detail when what people want is just to virtualize a workload (with similar simplicity to Docker) and have it work wherever the standard 'workload virtualiser' runs.
I don’t think it suits all teams and use cases, but for us it’s absolutely fantastic and without going down the rabbit-hole of cloud-provider specific tools and recreating half the issues it solves, I’m not super sure what we’d use.
I've already climbed most of the learning curve so YMMV, but as a team of one and dozens of WordPress, MySQL, and bespoke app servers, kuberenetes makes ops manageable so I can spend time on things that really matter.
Deploying new web apps is trivial, declarative manifests are easy to reason about, TLS certs are issued and renewed automatically (cert-manager), backups are cheap and reliable (daily GCP snapshots), making changes to the cluster via declarative terraform is a breeze, etc etc. No way I could manage all the ops without leaning so heavily on the core foundation provided by k8s.
I think that's the first thing with k8s: it all starts with an app that requires several physical nodes.
Well _technically_, sure, we could have run a bunch of those products on a single machine, but there goes your durability and the memory overhead on some of them was quite Hugh, and properly fitting them onto a single machine would have required more optimisation and technical skills than the devs I was working with had or were inclined to do.
Most of the value I get from k8s is the hands-off nature of it - I get slack notifications (prometheus+alertmanager) if anything is happening I need to address (e.g. workload down, node down, API not responding, etc). Otherwise I can safely ignore my cluster and know everything's good. Spinning up a new WP site takes 10m with backups, TLS, monitoring, etc built in.
You are absolutely spot on because this is how not to pass the behavioral interview for Engineering Manager.
After a week of almost full-time work, I threw in the towel. Admittedly, I also had to learn concepts like reverse proxies alongside, too, so I was by no means well-equipped to begin with.
Yet, tossing together some docker-compose.yml files and "managing" them with a Python script has worked very well. Kubernetes really scarred me in that sense, and I am healed! Also, Caddy has helped me in actually enjoying configuring the webserver.
I am talking about a homelab, a single server at home, for home use. It's much safer now, with Docker compose, because I understand it and I wrote the core exposed part's configuration, the Caddyfile, myself, manually. I know exactly what's exposed, and it's exactly right the way it is!
The remaining risk comes from the services themselves having security holes, but k8s has that very same risk.
Does mean that anything that upsets the ingress controller is an outage, but for experimentation, that's probably OK.
There are very strong financial incentives for every individual developer and sysadmin to adopt Kubernetes, regardless of the impact it has on the organisation as a whole. In a sense this is engineering reaching the level of corporate maturity of the sales department who will optimise everything for their commission regardless of the organisations ability to deliver it at a profit, or even at all.
Then that organization is doing a terrible job of aligning incentives. I'm guessing their pay structure isn't terribly merit-based nor high enough that people aren't constantly thinking about other jobs.
If this is about FAANG (your comment wasn't, but others were), perhaps part of this is exposing larger problems in many smaller orgs. (note: I'm ex-FAANG and happily so)
That issue is not endemic to Kubernetes, but rather to any larger system past a certain age, you learn stuff as you go along and would do stuff differently if you did it again today - but you can't easily, because you cannot break compatibility for everybody using your stuff.
As a concrete example from the Kubernetes world, there is a talk by Tim Hockin [1] about how today, they would fundamentally design the api-server differently and base pretty much everything on CRDs.
[1] https://www.youtube.com/watch?v=ji0FWzFwNhA
Also, I don't much like Go's templating syntax.
Not every addon/tool for the k8s ecosystem is worth it. I also don't bother with the ever-growing list of service meshes... not enough value to me for the overhead.
K8s is definitely the simpler alternative for me but there is still a lot of essential complexity in k8s due to the nature of the problems it's trying to solve. Mostly I like building on top of a solid foundation of standardized k8s API objects (pods, services, volumes, etc).
Tldr; Bring in only the add-ons and tools you really need so you don't add more complexity than necessary. Don't get swept up in the hype and marketing from other devs and cloud vendors.
Really when looking at tools in the k8s ecosystem, it's better to approach it as you would importing a new library into your application. Most decent devs wouldn't blindly import a new lib so that they can copy/paste a single line of code they found online for a business critical function, and k8s tools should be no different. We must think about what value does a given tool bring, and is it worth the cost of learning/maintenance? Sometimes the answer is a resounding "yes", but too often the question isn't even asked.
If you're looking to have your small app eventually grow into a large one, read up on K8s and just make sure you're not blocking future-you from making your app work on it. E.g., work well in a container (which is useful for automated testing, deps management, etc), have a simple 'ping' endpoint to make sure the app is up, have a better config story than "recompile to change these variables", use a logging library, and tolerate any other services you're using to sometimes be down.
All useful things for a grown-up app to do anyways, all a bit of a PITA, and all better than trying to operate an app that doesn't do them.
If you have one monolithic backend service (and most web applications really should start out this way), Kubernetes offers almost no benefits over alternatives.
Nomad is way simpler to get a cluster up and running, has a great configuration syntax (I'll take HCL over YAML anyday) and had first class Terraform/Consul/Vault integrations.
Onboarding devs is fairly straightforward, if they can write a docker-compose.yml, it's an easy transition to a nomad job specification.
It took me by myself ~4 months to get our current hashistack(Vault/Consul/Nomad) stood up using Terraform+ansible. Two members of my team have been working to replace the hashistack with a self hosted K8's deployment and they just went over the 1 year mark and we still do not have something capable of hosting the workload currently running on the Hashistack.
This got a little long winded but I feel like this "it's docker compose or K8's, take your pick" mentality had led to a bunch of needless time being spent by smaller teams/companies on solutions that just aren't right for them.
This feature helps a lot with that problem by bringing GCP closer to where AWS has been with Fargate. k8s will still be more work than using AWS ECS but it might also be preferable if you dislike using the provider’s components and want the control of, for example, doing your own load balancing and storage management.
Also you can get red/green deployments and rolling deployments with little to no effort, which can be very nice, nice to have.
IMHO, most of what K8s offers can be obtained cheaper and in a simpler way using more "traditional" DevOps approaches and systems. I'm currently operating an IT infrastructure consisting of more than 20 different components, using only basic Linux technologies and open-source packages (ssh, iptables, ipsec, ferm, (r)syslog,...) plus Ansible to orchestrate it all. Never encountered a problem that I wasn't able to debug and fix within a few hours, and managed to have more than 99.99 % uptime so far. I understand that this approach might not work for large companies, but it seems to me a lot of startups are going down the K8s route just for the sake of it, and their DevOps processes become incredibly brittle and slow as a result.
What I miss the most is having my infrastructure defined as code, instead of via the GUI. But given that I have only four services (out of which two use preemptible VMs and only one needs to scale) it’s not really a problem — it wouldn’t take me many minutes to replicate this setup at another cloud provider.