One of the things I hear the most about Kubernetes is the complexity. Honestly, it's not that complex from an user perspective. It's not much more complicated than Docker, really. It's also super nice to just write two yaml files and have everything running nicely with rollouts, auto-scaling, node pinning etc. It just works and I never have to even think about the internals (although I took the time to learn most of them).
Adding: heck, I'll run a single node k3s on my computer (which takes one (1) command) instead of using docker-compose just because I like it better than compose.
Yeah most of the (IMO) legitimate complaints about the difficulty of k8s are about how hard it is to correctly set up and administer. (Networking setup, making ingress work, cloud service integration, etc.) Especially when you get into permissions/namespaces and trying to get that right for a decent size organization and trying to configure things for least privilege. Simply using it isn’t that hard IMO.
If you need the scale provided by k8s, you are probably in a large org.
If you are in a large org, you are probably intermediated from running k8s directly by some kind of devops/cloudops team who needs to sort out all the "hard administration stuff" you listed.
Setting up a science project at home in your toy AWS account I'm sure is no big deal.
But this is true of every tech- "works on my box" is always easier than "doing this in enterprise".
The measure of a technology is how easy it is to use in the real production environment.
Being in a large organization does not equal having a dedicated team for running Kubernetes clusters. For example Daimler (the company behind brands like Mercedes) has hundreds of Kubernetes clusters across hundreds of small product teams.
IMHO, the biggest annoyance and difficulty with K8s, is figuring out the helm chart ecosystem with the crazy templated YAML. Some of those documents can get quite convoluted. One family of charts (say, from Bitami) will have one set of conventions for values.yaml, and other families will have a slightly different way of organizing this.
The problem is a lot of people read about k8s and decided stuff that could be done with a queue, 3 micro services, and 4 cores of CPU running on a RasPi somehow needs ... k8s.
This might be true, but one of the under-appreciated benefits of Kubernetes is the control around deployments.
3 services and a queue isn't much... unless you have high uptime requirements, need to maintain capacity during releases, want a careful rollout process, etc.
It's possible to orchestrate complex release flows without an orchestrator, but it's often easier with one.
I find consuming K8s to be like this - it's great, provides facilities for common everyday things
I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it.
It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it
edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking.
My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB
Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.
I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.
You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit.
The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.
> For a 'solution', you have to build a lot of it.
This is true even in the 'managed' k8s world (like EKS). They'll automatically configure EBS storage for you, but that's not sufficient for non-trivial workloads. Same with networking, logging, or metrics. I like that EKS means I don't have to worry about etcd backups and restores, but it's not a production ready platform by any stretch.
This is why you should never run plain-old-Kubernetes in prod (or in dev, if you want my opinion). Stock Kubernetes basically ignores security and, then, as you said, it requires a bunch of additional work to config.
There are several companies that take Kubernetes and polish it up, the two biggest examples are Red Hat OpenShift[0] (which I'm most familiar with, because I manage it as my day job), and Rancher[1]. Also, note OpenShift is a paid product, but Red Hat also releases OKD[2] for free, which is OpenShift without support.
OpenShift is Kubernetes with great default security policies. Storage, networking, logging, and more are all configured out of the box. I would recommend anyone who is using or looking to use Kubernetes to strongly consider OpenShift or OKD.
Could you share those 2 YAML files? I feel like it's not hard to do k8s YAML the wrong way and miss out on the auto-rollout, auto-scaling, etc. features
It moves most/all of the complexity on ops side. or in "whatever automation thingy you use to deploy it". Yes, easy on dev, but compared to "just docker" or "a VM with some systemd service running your app" it is hideously complex
Amazon publishing a blog that blatantly argues for vendor lock-in? Who would have guessed?
One of the big benefits of Kubernetes is the cloud-agnostic API for running pretty much any application. CustomResources can help blur that boundary and make it easier to use some managed services, but actually arguing in favor of that approach for everything is a pure advertising tactic.
It's no secret that managed services are several times more expensive than equivalent EC2 time. Sometimes it's worth it. But to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill. Running any Kubernetes cluster is not cheap, even if it's just a bunch of controllers.
> It’s arguably an extreme option and one that doesn’t even exist today
How is it blatantly arguing for this thing when it literally calls it extreme and something that doesn’t exist? I think a more fair criticism would be the fact they don’t call out vendor lock-in as a tradeoff
> to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill.
I think you have a valid point it may be dubious but the article does outline purported reasons it could make sense for some people. I dont think there is a “right way” and a “wrong way” here, rather there is the way “you would do it” and “the way they’re proposing as an option”
The author admits it's extreme, but is clearly trying to normalize it:
> Someone defines Kubernetes as “a well-designed and extensible API with programmable reconciliation logic, that happens to come with a container orchestrator built in.” I like that definition.
Disclaimer: I work for AWS but had nothing to do with this blog post (I'm seeing it for the first time with everyone else here).
I think this is an unfair summary of the post. Of course, using Kubernetes to orchestrate other AWS services is going to be a go-to example on the _AWS_ blog, but there is plenty of vendor-agnostic software doing similar things: DNS Records[1], Databases[2], even using Kubernetes CRDs to deploy Kubernetes[3].
The idea of using Kubernetes as an API to orchestrate external resources doesn't inherently lock you into any single vendor.
also on every worker server around 15% of RAM is dedicated to kubernetes pods (kubelet). So slightly higher price for RAM and CPU.
So basically overhead is like $100/month. Or $50/month with one master server which is not very reliable but acceptable for small workloads. If your project runs on $5/month VPS, Kubernetes is pure overhead. If you're paying $500/month, Kubernetes is something to consider.
I also ran a single instance k3s node on my personal $8/mo 3gb vps, and it took up around 400MB. Ran some workloads, a postgres operator. Ingreds via traefik on the node. I think my overhead was like $1.20/mo. Did I have fancy HA? No. But it was an easy way to manage workloads & learn skills. Works fine.
You're paying 4x as much for your self-managed cluster as I am for a vendor managed cluster. I have a single-node digital ocean cluster without a load balancer.
As I was reading this I thought it was calling for using the K8s API as an abstraction over all types of backend compute vendors. That would actually avoid vendor lock-in. Set up one control plane somewhere and use it to manage your AWS, Azure, GCP, on-prem VMWare etc.
That would mean you need the K8s API for your control plane but not necessarily that you have to run a K8s cluster or clusters as long as the API layer is the same. Maybe the control plane is a managed service too and you just interact with it.
I'm a bit skeptical of these single pane of glass type solutions since the abstractions always end up being somewhat leaky. The abstraction turtle stack is so high at this point you have to know 11 different technologies just to start a process on a Linux box.
But I get the operational appeal of having a central control plane rather than a million control planes all with their own AuthN/AuthZ, logging, custom YAML DSL, APIs, security configurations, integration points etc.
[ Disclaimer: Author of the blog here ] I knew this blog was very polarizing so this is a fair comment. I have called out advantages and disadvantages in the first scenario ("The pros of this approach is that the customer is decoupling heavily from the infrastructure underneath thus enabling better portability of their entire platform. The cons of doing this is that the customer is signing up for a high amount of work.") and, in retrospect, I should have called them out explicitly for the last scenario (I have just assumed readers could infer they would be the opposite). I did mention towards the end that "customers [that can benefit from the third model ] are coming from an Amazon ECS background, they are all in into Amazon ECS and AWS" but I take to heart the suggestion that I should have been more explicit. Thanks.
There has been a lot of prior work in this space. There's crossplane, the Terraform Operator, and a ton of roll-your-own solutions for tackling this vary problem. Alibaba also was early to have the concept of a 'virtual node' where you can deploy seemingly infinite number of pods, they'll handle all the orchestration below.
I was reading the article thinking "Yes, finally, some code or use to implement interacting with AWS without a 3rd party lib" and then I got to this section at the bottom:
> In this post, we showed you the flexibility of Kubernetes. It’s arguably an extreme option and one that doesn’t even exist today (the Amazon ECS ACK controller isn’t available but please thumb up this roadmap proposal if you are intrigued). If anything, it proves that Kubernetes can be a lot of things to a lot of different people and defining it remains a work in progress (at least in my head).
As usual, Amazon is playing catch-up in actually orchestrating their services in a non-AWS-centric way. Very deceptive to create a blog post and not disclose it's not real software at the end.
I love how conveniently on-premise kubernetes is missing. Or self-managed in the cloud.
The whole point of kubernetes is to not think about your servers anymore because they in essence can be replaced at any time. This would only provide absolute vendor lock-in.
Of course you can use any external service there is (like a database) but you should never put all your eggs into one basked and trust only one hoster. The principle remains the same: its just someone elses computer.
Storage and networking are solved in a lot of places at a fraction of the cost and that is really just the beginning., but also the whole point of the exercise: to be able to move your workloads from wherever to wherever
Terraform (for cloudy stuff), kickstart and ansible (for on-prem stuff) would be my first go-to(s) for automatic installation, configuration, and config management.
well, the old way of doing it is with named VMs "webserver01". The more modern way of for those names to essentially be UUIDs. They're individually unimportant, and get spun up/down automatically based on load and/or health.
so in essence making it harder on yourself to avoid using yaml?
The point of kubernetes is that its APIs all the way down, controlled by a central loop.
Its not that we can not know the uuids or names of containers, its that we usually do not even need them. Hardware problems are reduced to Node levels and workloads are completely replaceable. So in essence Kubernetes is a vendor neutral "vm server creator".
In any bare-metal scenario i prefer the approach to provision the cluster itself with terraform or ansible, because i am able to point at hardware itself, but otherwise the whole point of having multiple servers is to scale. I can also still have the same attachment to a lovely server without naming it peter. Stuff happens. at some point you need to take down a node or two. What happens here is up to you.
I also understand that everything is possible if you script it enough, but there again we are basically just talking to an API server for the workloads.
Even in traditional system admin vs cloud admin teams, each can focus on what they are good at in the same way your hardware should be separate from your software.
you inevitably do too.
The difference is your complexcity is usally done by hand and the complexcity in kubernetes only comes from your perspective.
Most of these controlling "algorythms" used in kubernetes literally only wrap the api of the underlying software, which in den end is the same you use too.
You too need networking, compute and storage just in the same way, but either you can orchestrate it or get orchestrated.
46 comments
[ 3.1 ms ] story [ 115 ms ] threadFor me it's a platform to deploy apps without tying myself into a particular vendor.
Adding: heck, I'll run a single node k3s on my computer (which takes one (1) command) instead of using docker-compose just because I like it better than compose.
If you need the scale provided by k8s, you are probably in a large org. If you are in a large org, you are probably intermediated from running k8s directly by some kind of devops/cloudops team who needs to sort out all the "hard administration stuff" you listed.
Setting up a science project at home in your toy AWS account I'm sure is no big deal.
But this is true of every tech- "works on my box" is always easier than "doing this in enterprise".
The measure of a technology is how easy it is to use in the real production environment.
https://kpt.dev/
However if you do need what it provides in _some way_, Kubernetes provides a pretty nice, coherent way of doing it.
3 services and a queue isn't much... unless you have high uptime requirements, need to maintain capacity during releases, want a careful rollout process, etc.
It's possible to orchestrate complex release flows without an orchestrator, but it's often easier with one.
I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it.
It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it
edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking.
My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB
Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.
The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.
This is true even in the 'managed' k8s world (like EKS). They'll automatically configure EBS storage for you, but that's not sufficient for non-trivial workloads. Same with networking, logging, or metrics. I like that EKS means I don't have to worry about etcd backups and restores, but it's not a production ready platform by any stretch.
There are several companies that take Kubernetes and polish it up, the two biggest examples are Red Hat OpenShift[0] (which I'm most familiar with, because I manage it as my day job), and Rancher[1]. Also, note OpenShift is a paid product, but Red Hat also releases OKD[2] for free, which is OpenShift without support.
OpenShift is Kubernetes with great default security policies. Storage, networking, logging, and more are all configured out of the box. I would recommend anyone who is using or looking to use Kubernetes to strongly consider OpenShift or OKD.
[0] https://developers.redhat.com/products/openshift/overview
[2] https://www.okd.io/
[1] https://www.rancher.com/
One of the big benefits of Kubernetes is the cloud-agnostic API for running pretty much any application. CustomResources can help blur that boundary and make it easier to use some managed services, but actually arguing in favor of that approach for everything is a pure advertising tactic.
It's no secret that managed services are several times more expensive than equivalent EC2 time. Sometimes it's worth it. But to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill. Running any Kubernetes cluster is not cheap, even if it's just a bunch of controllers.
How is it blatantly arguing for this thing when it literally calls it extreme and something that doesn’t exist? I think a more fair criticism would be the fact they don’t call out vendor lock-in as a tradeoff
> to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill.
I think you have a valid point it may be dubious but the article does outline purported reasons it could make sense for some people. I dont think there is a “right way” and a “wrong way” here, rather there is the way “you would do it” and “the way they’re proposing as an option”
> Someone defines Kubernetes as “a well-designed and extensible API with programmable reconciliation logic, that happens to come with a container orchestrator built in.” I like that definition.
I think this is an unfair summary of the post. Of course, using Kubernetes to orchestrate other AWS services is going to be a go-to example on the _AWS_ blog, but there is plenty of vendor-agnostic software doing similar things: DNS Records[1], Databases[2], even using Kubernetes CRDs to deploy Kubernetes[3].
The idea of using Kubernetes as an API to orchestrate external resources doesn't inherently lock you into any single vendor.
1: https://github.com/kubernetes-sigs/external-dns 2: https://github.com/kubedb/operator 3: https://cluster-api.sigs.k8s.io/
Overhead is as follows (/month):
1 x load balancer: $20
3 x master servers: 4GB RAM, 2 CPU, 15 GB SSD: 3 * $20 = $60
also on every worker server around 15% of RAM is dedicated to kubernetes pods (kubelet). So slightly higher price for RAM and CPU.
So basically overhead is like $100/month. Or $50/month with one master server which is not very reliable but acceptable for small workloads. If your project runs on $5/month VPS, Kubernetes is pure overhead. If you're paying $500/month, Kubernetes is something to consider.
That would mean you need the K8s API for your control plane but not necessarily that you have to run a K8s cluster or clusters as long as the API layer is the same. Maybe the control plane is a managed service too and you just interact with it.
I'm a bit skeptical of these single pane of glass type solutions since the abstractions always end up being somewhat leaky. The abstraction turtle stack is so high at this point you have to know 11 different technologies just to start a process on a Linux box.
But I get the operational appeal of having a central control plane rather than a million control planes all with their own AuthN/AuthZ, logging, custom YAML DSL, APIs, security configurations, integration points etc.
I was reading the article thinking "Yes, finally, some code or use to implement interacting with AWS without a 3rd party lib" and then I got to this section at the bottom:
> In this post, we showed you the flexibility of Kubernetes. It’s arguably an extreme option and one that doesn’t even exist today (the Amazon ECS ACK controller isn’t available but please thumb up this roadmap proposal if you are intrigued). If anything, it proves that Kubernetes can be a lot of things to a lot of different people and defining it remains a work in progress (at least in my head).
As usual, Amazon is playing catch-up in actually orchestrating their services in a non-AWS-centric way. Very deceptive to create a blog post and not disclose it's not real software at the end.
Paying for EKS just to use its API to provision and manage other AWS resources is an interesting idea, but not something I'd use in production.
The whole point of kubernetes is to not think about your servers anymore because they in essence can be replaced at any time. This would only provide absolute vendor lock-in.
Of course you can use any external service there is (like a database) but you should never put all your eggs into one basked and trust only one hoster. The principle remains the same: its just someone elses computer.
Storage and networking are solved in a lot of places at a fraction of the cost and that is really just the beginning., but also the whole point of the exercise: to be able to move your workloads from wherever to wherever
Isn't that also the case for VMs?
https://www.terraform.io/
https://en.wikipedia.org/wiki/Kickstart_(Linux)
https://www.ansible.com/
Your machines are still named. Your resources still multiplicative. Your setup is not idempotent.
Don't get me wrong, i have been using ansible and terraform for over a decade too, but this truly is an apple and oranges comparison.
Its not that we can not know the uuids or names of containers, its that we usually do not even need them. Hardware problems are reduced to Node levels and workloads are completely replaceable. So in essence Kubernetes is a vendor neutral "vm server creator".
In any bare-metal scenario i prefer the approach to provision the cluster itself with terraform or ansible, because i am able to point at hardware itself, but otherwise the whole point of having multiple servers is to scale. I can also still have the same attachment to a lovely server without naming it peter. Stuff happens. at some point you need to take down a node or two. What happens here is up to you.
I also understand that everything is possible if you script it enough, but there again we are basically just talking to an API server for the workloads. Even in traditional system admin vs cloud admin teams, each can focus on what they are good at in the same way your hardware should be separate from your software.
Most of these controlling "algorythms" used in kubernetes literally only wrap the api of the underlying software, which in den end is the same you use too.
You too need networking, compute and storage just in the same way, but either you can orchestrate it or get orchestrated.