For those unaware: Kubernetes has long had horrible default security. RBAC isn't even enabled by default. Nearly every cluster I've seen allows regular users and applications to breakout into the host nodes by default. Network ACLs, if they are ever used, almost never block either intra-cluster or extra-cluster attacks. Unencrypted API ports are standard. The distinctions between "cluster-level" resources and "everything else" unnecessarily complicates creating fine-grained rules. Default access even with RBAC is too wide-open. I won't even get into ABAC...
Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default.
I don't think it has ever been easier to take over cloud computing environments just by exploiting a dinky web app.
> Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default.
There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant.
It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.
Also, it doesn’t help that the kubernetes documentation is awful.
I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.
Well, there are two opposing approaches: "up and running" defaults or production defaults.
You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production.
With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run.
Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox.
K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".
Carefully manage complexity, and write a good manual.
OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does).
I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree.
> There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant.
> It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.
Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.
I had the same feeling, but frankly I didn’t bother replying because somebody putting sandstorm and kubernetes on the same level shows their complete lack of any understanding whatsoever.
Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production.
I worked with K8s professionally for several years but thanks for assuming I have no idea what I'm talking about.
Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.
Out of curiosity, which bits of the k8s documentation (if there are specific bits) did you find lacking? If it's security related stuff, I can open some issues to see if things can get improved :)
K8s documentation falls essentially in two categories: either reference page for k8s components (like the kubelet) that lists all the options woth a very very very brief description … or conceptual tutorial-like that explain some concepts, but without much details.
Nothing in the middle.
There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website.
The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.
Horrible scaling to tiny clusters as well. I'm glad they kept the design pure though. Unencrypted means you get to choose the encryption. Redis didn't have built in encrypted tcp servers for a long time either. When they added it there was a good discussion about the overhead: https://github.com/redis/redis/issues/7595
RBAC is now enabled by default. But I wonder how many of these clusters that are exploitable are Openshift for example. Openshift had secure by default settings from as long as I can remember. Non-root pods as default, api-server and even metric collection ports are protected by authentication.
Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).
What I don't like about standard kubernetes (kubeadm) is how hard it is to configure pods being secure by default, so developers couldn't just mount host directories into the pods. You would need complicated configs for that and nobody does that.
For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations to a namespace to activate it.
The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.
I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things.
I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking.
One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it.
There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it.
It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.
yeah if you give a user namespace edit access, then PSA is not going to help you (apart from that default config option you listed, but then even that is tricky as you need to know the namespaces to exempt like kube-system and others)
Kubeadm is a pretty barebones option, which I think needs work to scale (you're on the hook to add all the bells and whistles to create a full k8s environment)
Really for a more "batteries included" kind of cluster, you'd need one of the larger distributions that tries to do more out of the box. Obviously OpenShift is the ultimate example of that (for on-prem) but it massively increases the complexity of the environment.
Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.
Jeff in Engineering might have deployed an app spanning vCenter as well as EKS, and to continue to allow the app to function wrote subpar ingress/egress rules.
443 is just a port. If we assume that implies HTTPS then that’s still only TLS. If I exploit the service (“securely”!) I am inside your network. I can then pivot to your management layer. Game over.
I've seen non-tech corporations adopting Kubernetes without realizing that it's just one part of Google's infrastructure, not the entire operations side. There's an entirely separate chunk of infrastructure that provides the kind of security you mention.
K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.
Well, Google uses Borg internally which isn't exactly Kubernetes. But yes, the idea is that for it to be useful as infra it has to come without a lot of magic automatically attached.
Whilst Kubernetes defaults are indeed not great, they aren't that relevant unless you're building your clusters from scratch instead of using a k8s distribution, and I don't think I've seen anyone do that in a long time.
RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s)
Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common
Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now.
I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)
Well first of all, there are still thousands of clusters running with the old insecure defaults, so that's bad.
Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings.
The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS.
Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege.
Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev.
You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.
People not upgrading is indeed a problem but not one the distributions or k8s can particularly address, unless they force upgrade people which with deprecation/removal of APIs doesn't tend to be popular :)
The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point.
I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references.
for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.
I had a similar experience when I tried to use kubernetes in my home lab. Crashed it twice and since the technology and tools evolve so fast almost all tutorials I found were already so out of date that the commands didn't work and some tools were already deprecated.
I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box.
Do you go to the hot new restaurant in town on their grand opening, and then leave a review that says "the food was great, but it was too crowded and the service was slow", and never go there again? That sounds like what you did there.
The point of their comment, I think, is that a restaurant's grand opening is not always indicative of future experiences. The cooks/chefs, and front of house staff are all fairly new to the environment, menus might change as they find out that certain menu items are throwing off their timing or they don't have a great way to procure certain ingredients, and it will take time to all settle into a routine that works.
I think they were saying that is akin to the state of kubernetes, where everything is too new to really judge it today. Let me make clear my bias before I get to my opinion: I'm a big fan of what kubernetes has accomplished with managing the deployment of applications. But I think comparing current day kubernetes to a restaurant's grand opening is naïve. Kubernetes is amazing at what it does, and for how highly flexible it is. But it is highly flexible because of how advanced it has become, which in turn requires more knowledgeable administrators. You can take off with Docker Swarm because... it's very simple. It doesn't aim to be flexible, it just aims to be the docker-compose of distributed systems. Which is fine if you don't need the flexibility that kubernetes offers anyway.
And while I do appreciate kubernetes for all of its strengths, going back towards the point of this OP, I do believe it's a bit of a security nightmare. For example, the kubernetes operator pattern often just requires this application to run in your cluster with a ClusterRole granting it read and potentially write levels of access to ConfigMaps and Secrets around your cluster. Often without much in the ways of a method of telling the kubernetes API server what namespaces it is allowed to view those objects in. Worth mentioning though that secrets management is something that Docker Swarm also handles fairly poorly, and honestly isn't much of a security regression from the normal "deploy a VM that has all your decrypted secrets sitting on the system" anyway.
The description sounds a lot more like a trendy restaurant where no one can figure out what is actually on the current menu. That's more of a permanent complaint than an opening night one.
> What did you set out to do with Kubernetes to begin with?
Get some actual servers up and running to do real work vs getting certified on how to get k8s running.
You can't use trollish usernames on HN so I've banned the account. You're welcome to email hn@ycombinator.com if you want to use the site as intended in the future.
> I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box. I'll stay with swarm for now.
I've had decent luck in the past with the K3s distribution, which is a bit cut down (but certified) Kubernetes: https://k3s.io/ Others might also mention K0s, MicroK8s or others - there's lots of options there.
It also integrates nicely with Portainer (aside from occasional Traefik ingress weirdness sometimes), which I already use for Swarm and would suggest to anyone that wants a nice web based UI for either of the technologies: https://www.portainer.io/
But even so, I still run Docker Swarm for most of my private stuff as well and it's a breeze. For my needs, it has just the right amount of abstractions: stacks with services and replicas that use networks and can have some storage in the form of volumes or bind mounts. Configuration in the form of environment variables and/or mounted files (or secrets), some deployment constraints and dependencies sometimes, some health checks and restart policies, as well as resource limits.
If I need a mail server, then I just have a container that binds to the ports (even low port numbers) that I need and configure it. If I need a web server, then I can just run Apache/Nginx/Caddy and use more or less 1:1 configuration files that I'd use when setting up either outside of containers, but with the added benefit of being able to refer to other apps by their service names (or aliases, if they have underscores in the names, which sometimes isn't liked).
At a certain scale, it's dead simple to use - no need for PVs and PVCs, no need for Ingress and Service abstractions, or lots and lots of templating that Helm charts would have (although those are nice in other ways). Unfortunately, Swarm isn't awfully good for job prospects, which I have to admit. Then again, one can sort of say the same about Hashicorp Nomad, yet it's also alive and kicking.
It isn’t deliberately complicated, and once you know why it is designed the way it is, it becomes very capable.
The biggest thing to understand about Kubernetes is that you cannot understand it as a normal piece of software. You don’t directly control it, but rather, define the conditions under which a collection of automations respond to things going on. This is how it self heals. The whole system resembles more of a garden, where you are providing the conditions in which the plant grows, but you are not directly causing the plant to grow.
For example, if you understand a pod, then the next step is understanding how to keep a pod running, how to detect failures, what go do if it does fail, and how to change out one set of pods for the next. This would be a deployment.
You build your understanding by understanding more independent automations. For example, after understanding a deployment, you can look at how horizontal pod autoscalers work. An HPA knows nothing about a deployment and its internal state, yet will inform the deployment how many pods it should be running based on the metrics it sets. The HPA and the deployment scheduler otherwise work independently, and you can indeed write something that functions as an alternate HPA, and Kubernetes will do it.
There’s also an addon for cluster autoscaling. The cluster autoscaling knows nothing of deployments or HPA. It only checks pods that are being requested, but cannot be scheduled due to constraints. It then add nodes so that pods can be scheduled.
These three different processes operate independently, and don’t share state. They each watch for a limited set of signals and take limited set of actions. If any one of these processes goes down, the system can still limp along. Together, they create a much more complex behavior out of much simpler behaviors.
> It isn’t deliberately complicated, and once you know why it is designed the way it is, it becomes very capable.
I’m sure it isn’t and I changed the emphasis to make that line more about the slope of the learning curve.
Thanks for this explanation. The inability to grok the internals is really hanging me up. I’ll embrace the simpler “distributions” as a way to understand the higher level constructs.
Does it also make sense to ignore the fact that this is built on Linux’s containers and namespaces?
Based on my very limited understanding Kubernetes is basically a “cloud operating system”. This is why the underlying components like networking and container managers can be swapped out. But I still don’t grok how the subsystems fit together.
The pod is the basic abstraction. The underlying container technology can be swapped out. People have made Kubernetes schedulers for Microsoft technologies and WASM runtimes as well. If you wanted to use microvms, you can. Most people stick with Linux containers.
The Linux container/namespace is useful in the sense that operational characteristics matter, and many troubleshooting can end up going deeper and deeper in the stack. But as far as understanding how to _design_ things, knowing how the pure abstractions are composed together lets you be able to apply it to many systems, whether it is Kubernetes or something else.
I suggest reading through the first few chapters of Roy Fielding’s PhD dissertation. He is known better for REST, but his dissertation was actually about architectural elements, properties, and constraints, and then applying to hypermedia as an engine of state.
Once you know how to reason through computing architecture, you can pick apart any architecture, framework, or paradigm, and know how to work with it, and when you fight against it. You don’t grok a system by finding out how specific parts fit with each other, but rather, how things are consequences of the architectural elements, properties, and constraints. Then, the specific way things work (and don’t work!) together pops out.
And don’t be afraid to dive into the messiness behind an abstraction. Knowing and being willing to do that can set you apart in your career.
> What does it mean that k3s is “certified”? Should I care? What’s the tradeoff?
I think that maybe I should have linked directly to the K3s GitHub, which goes into detail about what it is, what the components are, as well as links to more information about testing: https://github.com/k3s-io/k3s/
Kubernetes is a large and complex system, but doing end to end tests aims to ensure that various implementations work correctly, they have pretty in depth docs on all of this, including conformance tests in particular: https://github.com/kubernetes/community/blob/master/contribu...
Well, you typically do need an ingress controller that will allow exposing services to the outside and direct network traffic.
Traefik is liked by some, less so by others, but generally works okay - except that its documentation (for K3s in particular) isn't as abundant as it would be when using Nginx. For example, if you want to use a custom SSL/TLS certificate (without ACME) for all domains by default, figuring that out might take a while.
That said, if you don't like it, technically you could swap it for something else, however I personally had a cluster hang once while doing that due to some cleanup action getting stuck.
However, do be warned that there are many different ways how to build clusters that might have significant differences.
K3s is just one of the Kubernetes distros - that attempt to provide an opinionated view on how to mix and match the components out there to end up with something that will let you use Kubernetes and have things actually work as expected at the end of the day.
A bit like how we have OS distros like Debian, Ubuntu, RHEL and so on.
I put stuff in systemd units and route them on HAProxy. For home network it's more than enough. Occasional container if app is designed so badly the authors can't figure out how to make it easy to install or package
A couple years ago I came into a company that had recently switched almost everything to Kubernetes. The company expected the devs to learn how to own and operate the cluster. Most of the devs there couldn't care less about anything except how to deploy their code to it (probably rightfully so). But I was shocked to find the k8s api was exposed to the internet and unauthenticated.
I brought the problem up with the higher-ups and they were pretty lethargic about it. Moral of the story I guess is if you're gonna choose to operate infrastructure like Kubernetes make sure you have experienced people trained to do so and don't expect your devs to wear another hat.
> Most companies that seriously deploy k8s tend to have dedicated teams for it.
Ya, and everywhere I've worked, that team is called "devops". The application developers are a different team. Maybe that's not the norm, but I thought it was.
I started back in 2016 building k8s clusters from scratch via terraform and bash scripting. It was ugly awful but worked thru Covid with zero downtime when we switched to hosted solutions from AWS and Google cloud.
There are app developers like that. There is a class of problems where you need to be able to do both application development and system work to be able to do well.
What a lot of people miss about larger k8s deployment is that you can create an internal platform team, like a micro-heroku within the company. The company would have to hire people who can create _products_ and also knowing how to work with infra in order to create a platform. It would be this rare combination of product, dev, ops, and focusing on supporting the rest of the engineering team and giving up on the glory of launching something for end users.
This is one of those cycles that IT does regularly every other decade.
In the early days, computers were huge things that were maintained by a team of specialists (and who actually soldered things together). Getting anything changed involved a series of paper forms and waiting for a specialist to do it for you.
Then came the microcomputer revolution and computers became small things that sat on desks and were maintained by the office geek. Changing anything meant talking to him (it was always him) and it would be done by lunchtime, and you owed him a coffee.
Then companies learned that this meant that no two offices had the same configuration, application suite, or even operating system. IT was centralised, and every office issued with the same stuff. Getting anything changed meant filling in forms and waiting for System Administrators to get around to it.
Then Agile and the Cloud happened and while the rest of the company had to carry on filling in forms, the IT Department turned into Engineering and DevOps was born. Every developer learned how to deploy their stuff to the cloud by themselves.
Then organisations worked out that that meant that applications were deployed badly, using a wide variety of tools and technologies, and essential security practices got skipped because no-one was responsible for them. So DevOps was turned back into SysAdmin and the forms reinvented as JIRA tickets.
And so we go on. There are micro-cycles within these ones, like the one about terminals (are you typing on the actual computer, or on a terminal that talks to the actual computer which is somewhere else? The answer depends on which decade you're in).
The ideal solution is somewhere in between (enough central control for it not to get into a mess, but enough decentralisation to prevent bureaucracy and allow for agility in the actual business). But that is a very hard balance to find and keep.
DevOps still must draw the line somewhere. DevOps people typically don't physically plug cables into switches, for instance. With the ever-growing complexity of infra, I see that line having to shift more and more, with either PaaS products or platform engineering teams filling the space by providing something like Kubernetes as a service to the DevOps folks.
It usually doesn't work that way, in my experience. My title is "Senior DevOps Engineer" and I do my fair share of writing code as well as maintaining infra. But honestly, hiring for people who do what I do seems to be very hit and miss. Most people excel at one or the other, and so they only pick up tickets that fit their skillset. They might dive into a ticket outside their comfort zone here or there, but not commonly. It would seem to me that many DevOps engineering teams are just one team who has a manager that hired and manages some Devs and some Ops folk, and just wrangles each of them into doing tasks that fit the goal of the team and their individual talents. And from an outsider's perspective it would seem like one team of jacks of all trades, but internally it's very much known who is going to take on certain kinds of tasks and the siloes are all very much in place.
That situation sounds reasonable though. I don’t believe in tearing down all silos. Expertise and preferences exist, it’s human nature. I always thought the DevOps mantra of having one person be good at both was a fruitless endeavour. Have Dev and Ops collaborate closely, yes, but let them be separate people (not teams!). It’s fine.
You know how to dev and ops, and I don’t doubt it. I do too. But if we focused that same energy and expertise, we’d be double the devs/ops. Nevertheless, people like us are excellent glue for organisations. At the same time, teams and organisations of only us would be quite pointless.
I feel similarly about fullstack. You’re a front end dev that has written a controller and used an ORM before. Often not too inspiring. Backend devs are still needed, whereas full stack devs form excellent glue.
That’s what DevOps is… or, was. I remember when it was new; that was the whole thing: dev and ops should work together. A break from the norm of the time that dev throws it over the wall to ops and never thinks about it again. What a tragedy that it’s been cargo-culted into its current meaning of “devs do ops” (poorly).
You’re right. And it wasn’t my intent to editorialize (to imply I feel one way or the other about the subject.) But I can read how an opinion could be inferred. Anyway, my actual opinion is that people like us tend to be more useful than not, but also it definitely takes people with more specialized knowledge to make progress with some organization goals. For example, at my current role we have a team of people with a much deeper background in mathematics (we’re in the financial sector), algorithms, and computer science in general. Due to a life event, I dropped out of my computer science degree towards the end of my second year, and ended up pulling myself through operations roles (which I typically found comparatively boring) until I landed in DevOps after a lot of open source work and networking. But I am not as strong as some of my peers in mathematics, algorithms, and computer science topics (what is a big/little endian lol?)
I came in to my current gig as the first devops hire. We were a lot smaller then. 8 years on, I'm managing an infrastructure team alongside ops and devops teams. We also draw lines a little differently; infrastructure manages core services like auth, logging, instrumentation, etc.
In my experience, devops folks usually come from the dev side, not the ops side, and their experience and curiosity reflects that. So we let them focus on shipping line-of-business code while making sure what they ship runs in a competently managed environment, as well as handling data centers and all the more traditional infra stuff.
my experience hasnt always been this, but generally speaking, this is the way to be successful - you need developer-oriented people doing the work. in some ways its harder than coding, coding has more structure than devops automation
That's what I've been saying all along. For me, when I use something I need to understand it.
I'm usually a quick learner (3rd iteration is where I start making sense of a complex topic).
I beat my head against k8s for 2 weeks, every day, I wanted to setup a 3 node control pane on my own, and it was just too hard. I gave up.
Also, I don't have many resources at my disposal, k8s requires a separate storage cluster. All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage.
Way too expensive.
And then think further, all this has to be updated sometime.
I don't have the brain and time resources to do that.
So, if I want k8s, I need to pay for expensive cloud.
Or, I can remain with my 1 server with backups and build systems on demand.
Yes, I hear the booksmarties scream "but but failsafety".
1 incubator server, specialized systems when something takes off.
Some companies can afford to pay for expensive cloud, more power to them.
I can't. I'm busy solving problems at hand, not dealing with additional problems of a system I don't fully understand.
Yes, k8s is a nice comfortable way, if you can afford it. But it's expensive and wasteful of resources. And we do have a climate crisis.
> All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. … Yes, k8s is a nice comfortable way, if you can afford it. But it’s expensive and wasteful of resources. And we do have a climate crisis.
You can run each of those roles on the same machine, achieving high availability from 3 total. For learning purposes you can run this all in VMs on a single machine.
There are plenty of good reasons not to try kubernetes when it doesn’t fit your use case, but the overhead it imposes is mostly in time and expertise. Claiming it’s more costly in compute resources, and especially contributing climate crisis, is reaching.
Someone is building a systemd based kublet replacement.
Why? Because Kubernetes has some great ideas on self-healing and resiliency.
But if you are small, and can work with a single instance for availability, then probably should stick with systemd. You could also use podman to get a small bit of the kubernetes ideas for working with containers.
The next level of scale wouldn’t be the full k8s but more like k0s, or k3s.
> Someone is building a systemd based kublet replacement.
I'm working on a product that uses something like this internally, basically, we provide multiple preview-environments in the same big server, these are managed by systemd.
This is the first time I get to think that this may be a project by itself, ping me if you have any ideas or want to get into a discussion on this topic.
You don't need 3 control planes. If your control plane will be down for some time, your cluster will be OK. Even if your control plane will be down for prolonged time, your small cluster likely will survive. 3 control planes is good, but if you can't afford it, it's not necessary. Just make sure to backup etcd and keys for your control plane and learn how to recover it in case of hardware failure.
Kubernetes does not require storage cluster. If you need it, you can run it. But it's not required.
Please think about your arguments when comparing kubernetes with any other solution. If you don't need storage cluster with any other solution, then you don't need it with kubernetes.
Kubernetes at its minimum is 3GB RAM and 0.5 CPU for one control plane. That's all you need. Everything extra is not necessary.
People often try to build some kind of super-resilient cluster with kubernetes, which is suitable for huge company with hundreds of servers, while comparing it with one host docker compose. If your task is simple, don't make it more complex than necessary. Kubernetes can be simple or it can be complex, it depends on you.
What you do need for kubernetes is knowledge and experience. I spent several months tinkering with cluster before I dared to deploy something valuable there. Although I used kubeadm, with cloud offering few things are simpler.
> If you don't need storage cluster with any other solution, then you don't need it with kubernetes.
That is only half of the truth. If your application needs a database, and which application doesn't, you have to store its data somewhere. So you either need a storage cluster, or you open the can of warms called local volumes and (anti-)affinity rules.
For smaller application deployments, hosting a separate kubernetes cluster is extremely expensive in terms of time spend to debug infrastructure issues. k8s introduces orders of magnitudes of additional complexity to networking, dns, storage, routing, logging, etc. All in the name of making these exact same things easier for the developer.
This can be a good separation of work, if you are large enough to afford a dedicated k8s maintenance team. But often it is not.
Keep your db outside the cluster. Use a managed db or just host it on its own server. You should be doing that even if not using k8s anyway. Don’t share memory and cpu with your db.
Regarding storage, try to build stateless apps that only need temporary storage that don’t need to outlive a request (like uploads). For long lived unchanging assets, you can try including in the container (ie. defaults.json, countries.json, samples.json, etc). For assets that need to be fresh, try to have the app generate at startup (eg. Email vírus definitions) or via init container. I think this will cover most use cases.
This won’t work with apps that depend on shared storage like Wordpress, but Wordpress is horrible and I can’t support you running it :-)
I'm not sure I follow you about local volumes and affinity rules. I followed instructions on kubernetes website to create local volume and pod which uses this local volume was scheduled on the proper node every time. I think it works with volumeBindingMode: WaitForFirstConsumer option.
So if you think that you need storage cluster either way, go ahead and deploy it. If you think that you're OK with single unreplicated database, just use local volume, it works. At least I've yet encounter any worm with this thing. For me it just worked.
Another option is to use local volumes and replicated database setup. So you can run, say, two database instances in a replicated mode on two servers, but storage will be simple local volume. It provides some reliability and availability benefits without complex storage cluster solutions. I used cloudnative pg operator for this and it worked for me. I guess one can configure ordinary postgres replication without any operator, I just don't have experience with that.
My opinion is that even single-node kubernetes cluster is worth it. That's because I have some experience and won't need time to learn basics. If someone does not have that experience, then I agree. Deploying kubernetes requires learning curve and that time might be spent for other things.
At least it's hard for me to imagine any kubernetes complication that won't arise with, say, docker. At the same time, kubernetes provides lots of polished software line ingress-nginx, cert-manager, cloudnative-pg which could save some time. And, most importantly, using kubernetes from the start, provides very easy option for moving into cloud, for scaling, for zero-downtime deployments. Those things are valuable by almost everyone.
We went through these loops where I work due to the "you have to use k8s now" thing.
Your control plane and worker node can be the same machine.
Rancher + Rancher Kubernetes Engine made it a lot easier for us to manage as maintaining that is simply "run this docker container" and it deploys all the k8s gubbins for you. And k3s is a lot more lightweight on resources than k8s. Stuff like certificate rotation and kubernetes version upgrades are handled for us, it's reduced to clicking buttons on UI.
Storage can be local, it's just not / wasn't well explained in the documentation. PersistentVolume[Claim]s can use directories on the host, those PVs/PVCs can be shared between many pods and you can mount subdirectories into each pod using `subPath`.
You can get it all running on a laptop using a couple hundred MB of memory, all in all. So, not going to be ideal for running on a Raspberry Pi, but on par with a lightweight linux desktop environment in terms of resource usage.
I hate the complexity sometimes, but then I still end up having the same learning curves with things like SystemD when I use them anyway - every "run a bunch of stuff on my machine and restart it if it breaks" system seems to be rife with edge cases and behaviours which you have to dig deep into the documentation to see - and we don't _want_ to RTFM if we don't expressly _have_ to :-)
^ Although, okay, SystemD doesn't require you to have random HTTPS authenticated endpoints to talk to it and TLS certificates which expire every year and consists of _one_ init process rather than like 7 interdependent docker containers which can restart at any time and rely on container infrastructure to be running and argh. Okay yeah the flashbacks are happening and I agree with everything you've said again.
The bigger issue is that container deployment and management is a big reseller business and they actually don’t care about whether it’s appropriate or not or if the eventual maintenance and DevOps are in place.
disclosure: i am a maintainer and the software overlay in the middle (helps enforce outbound-only, pre-authorized connects only) needs to be managed (self-hosted foss or hosted saas), so there are still trade-offs.
I increasingly feel like the development community maintains an equilibrium of "Accidental Complexity" and "Essential Complexity". As better methods for managing Essential Complexity come to the fore, Accidental Complexity gets jacked up to keep the workload high, so the geeks stay stimulated and the consultancies collect the $$$.
Between mind-blowing complexity and never-ending distractions, I find doing actual work near impossible. After getting laid off, I just... didn't look for work.
I instead started writing my own product for pull request assignments and direct Slack notifications for those (my personal frustration).
I needed to get away from it all and decompress. It's been refreshing writing code.
I have been feeling this for a while now. It’s like the army of eager labor, the ranks of managers who derive value from larger work forces, and sick amount of VC money, has really eclipsed emphases on various axes of efficiency seeking that used to dominate the industry (often in overrated/oversold ways admittedly). Remember when “tech stack X enabled you to do y per coney more with z fewer developers?
In VC-backed startups, "growth" does not refer to user growth or profits. Most of the "daily active user" metrics are bullshit. I've worked for a company once that gave away employees free funding money to make purchases. Yes, to get the number of orders up before a board meeting. It was blatant, unethical, but not really illegal.
So what is growth? The number of developers you have doing stuff. You hire more and more, and every few months you parade some Patagonia vests through your offices to show them. "Observe, more devs! Growth!"
The problems is, the devs all have to do something.
I stopped feeling productive years ago. It's all just never-ending grind, dealing with complexity that has no f--ing business being there. And the younger engineers have no idea. They think this is normal.
Out of the tar pit suggests we should all go back to one big SQL server and probably resort to something approximating SSR web apps again. You know... 2-3 big powerful servers like we used to have. How fast is a stack like this today compared to 20 years ago?
This is mostly what we have done. We want to make money. There are barely 10 of us. We don't have time for make believe work that doesn't bring in extra revenue. We used to play in traffic with conversations like "micro services" and "one repo per service" but our investor killed that mode of business and gave us one more chance to do it right.
I recall the original reason for moving away from one big database being something approximating "it won't scale". How well are those various k8s contraptions scaling relative to the "legacy" verticals they purport to replace? How many of us would go back in an instant if given the opportunity?
I think the biggest fear for most on HN is going through ego death over this. The fact that even your "hard" problem is trivially solved with boring tools that were invented before you were born.
I wonder if even the multiple servers and dedicated SQL server are overkill for most applications. The simplicity of running the whole thing on one machine using SQLite, with Litestream or something like it for backups, is really seductive. Edit to add: and as you've pointed out before, the locality of having the database right there in the application process leads to really good performance.
Kubernetes (and many popular distros) allowing anonymous access, even when authentication is enabled. The system:anonymous user is generally not authorized to view resources, but might be inadvertently.
To test this, if you hit the K8s API server unauthenticated, do you get a 401 error or a 403 error? If the latter, you're at risk.
I didn't word my statement well.
Yes, you can enable unauthenticated requests and you can grant the anonymous role access to the api. I didn't recall `--anonymous-auth` to be true by default.
Most of the major cloud distros (AKS being the notable exception) do have --anonymous-auth enabled, although it's generally just /version and a couple of other endpoints exposed.
Makes it easy to find out what clusters are running what versions of k8s which is interesting, but not a major security issue.
It did in the old days of the "Insecure API port" which listened on 8080/TCP, but that's been gone a while now.
Whilst it usually was just bound to localhost, I did encounter a distribution that, by default, bound it to the container network, meaning that anyone with access to one pod, got to be cluster admin!
Generally speaking though I don't think any distributions do that now.
This is super interesting. If you actually look at a Loadbalancer logs without a WAF in place, you will see huge amount of bot activity.
What is not clear in the blogs is how did they actually accessed the API server without a valid service token. Secondly, they index alot on IPs, that implied these are k8s cluster that are not managed (EKS, GKE etc).
I am actually a bit surprised there are any clusters out there exposing their service ports. Do people not run firewalls in their nodes? Sure, it may cause some headaches due to a range of random IPs and random ports that Kubernetes expects to be able to bind, but I assume any sysadmin will know how to deal with this. I know k3s has the awesome advice of disabling firewalld[1], but presumably the clusters mentioned in this article are largely using kubeadm or some other tool rather than k3s.
With that said, I agree that secrets management on k8s is pretty bad (unless you use an external service like Hashicorp Vault, a frequent pattern that I run into with k8s). Even with RBAC and secrets in a namespace being inaccessible from other namespaces, the moment someone has permissions to read secrets from a namespace, its essentially game over for you, as noted in this paragraph.
> Another potential attack vector within a Kubernetes cluster is accessing internal services. ... In cases where authentication is required, the credentials can typically be found within the Kubernetes secrets space.
The firewalld is for the trusted zone… and for the apiserver 6443 https port k3s comes with rbac enabled by default so if you want to access it via kubelet you need to open it?!
trusted zone is used by overlay networks or node internal stuff (or if you manually put your interface into it)
Why would you be surprised? In most cases the only thing users need from a Kubernetes cluster is to access it from the outside... like... why wouldn't they expose it to the outside world? -- this is the most useful thing you can do with it.
And the range of things you might want to expose... oh, god only knows. Think about such trivial stuff as FTP or iSCSI -- kinda useful, right? -- but Kubernetes rules for port exposure are idiotic, because you cannot expose ranges. And quite a few of popular Internet protocols assume they can allocate whatever port they want and start talking on it. And then you start fighting Kubernetes... and you'll win, but it will be some custom security solution... with the obvious problem of it being a custom security solution.
Now, add to this that Kubernetes will typically mess with your iptables and maybe also arptables, and if your firewall is based on eg. iptables, then you might be out of luck using it with Kubernetes, and again, will end up rolling something "special".
Obviously, you might also want, say, DNS to work inside your cluster, but that's a separate can of worms.
This is also why managed Kubernetes is a useful thing (EKS, GKE, et al)... but if you still want to do it yourself, maybe look into some Kubernetes distros (like Typhoon (https://typhoon.psdn.io) which I run on my clusters)
But managed is expensive.
You have to pay at least double vs unmanaged (in the dedicated world, ofc there is no "unmanaged" k8s offer from those cloud providers, just an analogy).
You are then bound to that cloud and its traffic prices and all those you mentioned are filthy expensive when it comes to traffic.
If you are dealing with the level of scale that allows you to fully take advantage of K8s, the cost of managed offerings is less than that of deploying and managing Openshift on-prem.
Also, the beancounters don't like on-prem as much anymore due to the upfront cost outlay.
I've been working with a number of major enterprise accounts and they've all been adopting managed K8s offerings for this very reason.
I mean, it’s not the best argument to say a system is “one action away from disaster”. If you pull the escape lever in an F16 for no reason you’re also pretty screwed. If you intentionally fly into the ground no one blames General Dynamics for making the plane too complex.
It turns out complex tools for complex tasks require some skilled operators. Yes kubernetes is overkill for your home lab, no it’s not overkill for enterprise-grade AWS orchestration with hundreds of apps and millions of users. News at 11.
I want to host a website not bomb the Middle East. Most enterprise apps are still websites that need no more than elastic beanstalk. If your analogy is to compare k8s to f16 then maybe you are not doing your argument any service.
If your dealing with dynamic scaling needs K8s are quite nice. There's a reason why most Telcos have deployed 5G functionality using K8s.
If you are in a situation where you want to follow a decoupled/microservices development pattern (ie. split component ownership across dev teams) as well as manage scalability during high load periods, K8s is hard to beat.
K8s is a useful tool, but can be overkill in a number of situations. Just like a hammer, you can't use it to solve for every usecase.
P.S. F-35s use K8s clusters (or might be K3s) to deploy their EWS.
I've spent my entire career being on-call for web servers and Elastic Beanstalk is not enough. Deployment failures, deployment speed, deployment safety, upgrades, local testing... There is a reason AWS themselves is pushing EKS and not Elastic Beanstalk, and it's not because Kubernetes is trendy, it's because it's incredibly useful.
I’ve had production SaaS apps running on elastic beanstalk just fine for years now. Better SLA than any of the kube apps I’ve seen. You’re right that if you have highly capable support team and the real need for it go with the pods but if you just have mediocre engineers and a straightforward deployment with no real scaling issues then just stick to simple systems.
Sure, then don’t use k8s. If you need extremely resilient leaf node architecture or high volume communication from distributed machines, then that’s a good time to start looking at k8s. Your lack of charity reveals your naïveté: the analogy is actually a good one because not all software development is equivalent to hosting a website.
Kubernetes was built with no program for how to do important things. Every important thing in Kubernetes is post-hoc, poorly bolted-on patch instead of real solution.
Take for instance upgrades. Don't like upgrades? -- Try downgrades? -- Oh, what do I hear, you seem to not like that either... So, how about user management? -- Also no? -- Well, how about backups and recovery? -- What, again?
Kubernetes solved some easy problems quickly with no program for how to solve hard problems. Security is one of those hard problems Kubernetes has no idea what to do about. The reason? -- Kubernetes, on its own, provides very little. It doesn't know how to do networking -- you need a CNI for it. But there's no good interface in Kubernetes to allow the admin to somehow control in general how the cluster is supposed to behave. Maybe some container somewhere will do something tricky, and will punch a hole in whatever firewall you though you've built around it. And you cannot make sure upfront that's not going to happen. There isn't a single way to punch that kind of hole... and the problem is, often time simply to make things work Kubernetes users will punch those holes. Maybe because they don't know how to do things better, or maybe because there isn't really a better way, or the better way is so tedious and convoluted that nobody will really follow it...
It's unreasonable to expect anyone to be able to deal with this kind of complexity. We need to be able to manage complexity, but Kubernetes gives us no tools to do it. It's the same type of problem as Dijkstra described with "goto" -- it's the tools fault. It creates so much complexity that humans cannot cope. Not only humans, computer programs cannot cope -- too many things to check, to many possible problems to protect yourself against.
I disagree with everything you've said but there isn't enough content to any of your complaints to respond meaningfully.
Upgrades and downgrades work fantastically. Putting "a firewall in front" is trivial - just... put a firewall in front. I don't know what "punch a hole" means, but if you mean "bind a port", yes, there are indeed a lot of ways to do that (on linux, windows, and osx!), and kubernetes provides excellent, organized, standardized and controllable ways to do that.
Kubernetes does not create complexity in my experience. It's the other way around - developers who have never seen the complexity in the world complain about the complexity embodied in the tools. They think "you're not gonna need it", despite an entire industry of operators saying "actually yes, we do need this, and badly".
I've seen developers who've never built a website in their life complain about React, and it sounds exactly the same as this. You might not need these tools, but they're not unneeded. If you can make a simpler and more powerful tool, please, go ahead, you'll be rich and famous, and everyone from the US Navy to telecoms to where I work will celebrate you.
Just one example off the top of my head: ClusterRole vs Role. Two kinds of role. Difference? One applies to a namespace, the other applies to all namespaces, cluster resources, and non-resources. Why not just make "Role" deal with all of them based on metadata or annotations? Because fuck it, that's why. Oh, and permissions cannot include a "deny" section, they can only "allow". Why? Fuck it, that's why.
You can create a secret, but only in one namespace, and there's no way to automatically share it with multiple namespaces. Why? Fuck it, that's why.
K8s is a shitshow of both complexity and a lack of functionality. It's a design-by-committee atrocity that is so bad it created an entire ecosystem to fill in all the bullshit it doesn't do.
Worth noting that, on their own, namespaces are not security boundaries, they're (IMO) primarily intended to be organizational constructs.
What namespaces do allow for, from a security perspective, is for various controls (e.g. RBAC, Network Policy, Pod Security Admission) to be applied to a given application or group of applications (depending on how you've organized your namespaces) in a relatively controlled fashion.
You can disagree without ad hominem. Your points make little to no sense, even beyond your sneer.
Pods can’t be “upgraded” individually becuase pods are ephemeral.
Downgrades work fine - replace newer binaries with older binaries.
Upgrades work great too - replace older binaries with newer binaries. Single node is the correct context for upgrades because a node is the failure domain in kubernetes (and in every single other Linux based orchestration system). 1.28 even includes an automated way of doing this in a highly available way per host.
CSIs and CNIs are binaries, which can be upgraded and downgraded like with any other system. Yes, if you upgrade or downgrade major versions there may be breaking changes.
I’ve been a sysadmin and a devops engineer and a cloud platform engineer for approaching 20 years. I’ve flagged your reply as I don’t think it’s appropriate, but I wanted to reply to you anyways because I’m sure beyond your anger and rudeness and lack of empathy, you’re intelligent enough to do a bit of reading and double check that you’re not absolutely wrong on all counts.
At some point you’ll be surrounded only by what you think are idiots, and they’ll exclude you. At that point you might investigate if just possibly you’re not by far the smartest person on earth. It’s possible other people know what they’re doing too.
You probably know about resolv.conf, but you might not know about "search domains". A feature that should've never existed, but was added because "why not?". If you don't know what it does, in first approximation, it creates a domain alias. This is sometimes used to save some typing when using DNS for service discovery in larger clusters in a very hierarchical way (when names start to look like a.b.c.d.e.f.g...com, it's tempting to alias them to something shorter).
So, long story short. Ubuntu LTS had a bug (actually a whole bunch of them) in the same script that was responsible for generation of this file (it's a bad idea to have this file to begin with... but hey, who cares, right?). So, one aspect of the bug was that a variable name "DOMAINS" was used instead of its value. And that became a search address in resolv.conf.
Now, this search works by appending the search path to the domain name, if DNS server tells you it cannot find it. I had no idea DOMAINS was possible to register as a valid domain name. I mean, it should've been a TLD to be discoverable globally, right? -- Well, it's still a mystery to me, but, for some reason domains of the form a.b.c.DOMAINS are resolved to some AWS EC2 instance...
And, the punch line: if you screw up CoreDNS configuration somehow, or something else goes wrong with service discovery inside your Kubernetes cluster on unpatched Ubuntu -- there's a good chance your traffic will go to that EC2 instance. Who's on the other side -- I don't know. I didn't send anything important that way and discovered it because something like Spark operator didn't work... but it could've been much worse.
Oh, btw, I believe resolv.conf is used by different CNIs. Maybe Calico and Flanel. I cannot remember atm. why it was used... or maybe it's some kubelet setting.
Anyways. Larger point is... large Kubernetes clusters are virtually impossible to systematically audit. Too many things can do too many things. Too much trickery goes on in all sorts of system components that deal with networking. It's also impossible to track who does what and who owns what because there's no user identity.
---
I believe that, again, industry adopted an absolute horror show of a program / environment not because it was a great tool, but because "organically" developers started to add "features", creating something that's hard to resist, because, superficially, it creates a lot of value with minimal effort -- but it also lays a trap that is not immediately obvious. And those who find themselves trapped, instead of running away, star adding more "features" to fight the trap. But, in this foolhardy attempt, they only make the trap stronger.
I'm not familiar with w/e issue of Ubuntu caused `search domains` to end up in your resolv.conf. I'm skeptical it wasn't just a misconfiguration. At any rate, you had a problem with your host configuration.
The kubelet will merge the hosts resolver config with the cluster config to generate a resolv.conf file for the pods it runs, resulting in the issue you described. This can cause other annoying dns issues. It has nothing to do with CoreDNS. The only way I've found to disable this kubelet behavior is to start the kubelet with `--resolv-conf=/dev/null`
I'm replying to your comment, which brings up a lot of tangential issues about Ubuntu and AWS, none of which are specific to Kubernetes. As other replies have pointed out, "you had a problem" and the bug ticket you linked doesn't even mention k8s. Skept-o-meter over 9000 indeed.
I feel like the jump from "obscure corner case + bug in some script led to bad consequences" to "k8s is an absolute horror show" is not really warranted.
Servers are so much faster than they were ten years ago that a typical, fully-loaded enterprise app can almost always run on a single mid-sized 10-core box with 256 GB of RAM.. from ten years ago. It's absolutely ludicrous how much power is available now on a single box if you really want to scale up instead of out.
(Databases, of course, always like to eat everything because of poorly-optimized statements, but they've never really been good fits for K8s or containers anyway.)
> a typical, fully-loaded enterprise app can almost always run on a single mid-sized 10-core box with 256 GB of RAM.
If you don't care about high-availability, then yeah. One thing I've learned in my career is that delivering reliable enterprise-grade service is anything but trivial. The level of complexity of any modern stance is high enough that there will inevitably be sharp-edged corner cases which aren't simple or cheap to deal with.
Don't most outages happen because of unforeseen consequences of changes to the application/environment rather than machine failure? Is reducing the exposure to machine failure really worth exposing yourself to the vast complexity of distributed systems?
I think the point of such tools is more than just reducing exposure to machine failure. Reducing exposure to machine failure also means that you can run your hardware closer to end of life without having to worry about data outages.
The article about stack overflow mentions this little tidbit.
> That application handles around 6,000 requests per second, and around 2 billion requests per month.
To be able to handle loads of 1M requests per second, your going to need more than one server dispensing results. Especially since only so much data can be stored in memory without running into performance issues.
That said, these kinds of tools are an unique use case. Not all programs need them.
Why is this argument commonly used to dismiss kubernetes?
K8s is a way for you to drop a large number of such pieces of software onto a small number of such boxes, and keep the configuration and lifecycle of each of those deployments independent.
Kubernetes is not primarily a scalability tool it is an availability tool.
I've done quite a lot of looking at exposed k8s clusters, and one thing this piece doesn't really cover is what percentage of the 350 clusters they identified were in fact honeypots.
You can find k8s clusters without auth using Censys as they specifically try to access things like pod lists, and from spinning through lists of those they tend to be obvious honeypots (e.g. lots of ports that wouldn't usually go together being exposed), although I have seen the odd cluster that looks a bit more realistic, unlike Aqua I didn't try accessing it directly as that could be dodgy legally :D
While this post has some merit, all of these problems can easily be solved and there are dozens of guides on hit.
The entire "CIS Hardening guide" will give you step by step tutorials on how to solve all of this, you just need to bother looking it up.
Look, you need a trained sysadmin anyway, no matter where you deploy. See for instance all the ubuntu machines out there that never have seen the admin update it.
Or all the terraform scripts that simply get copy pasted by orgs.
While I do agree, this is a problem, there are security conscious distributions if kubernetes out there like RKE2 or "rancher government" to make things much more secure.
This is actually my job. If you need someone how to secure your cluster and make sure stuff like this is very unlikely to happen, you can find the info in my profile. But what it comes down to is that if your DEVs have direct cluster access and you have no monitoring and alerting setup, you are kind of operating on the same level as copying and pasting stackoverflow answers and pushing them directly to prod without code review.
155 comments
[ 2.1 ms ] story [ 208 ms ] threadBasically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default.
I don't think it has ever been easier to take over cloud computing environments just by exploiting a dinky web app.
There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant.
It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.
Also, it doesn’t help that the kubernetes documentation is awful.
I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.
You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production.
With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run.
Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox.
K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".
Carefully manage complexity, and write a good manual.
OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does).
I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree.
Sources:
https://openhub.net/p/kubernetes https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_li...
Edit: Using open hub for both says OpenBSD is 75M lines:
https://openhub.net/p/openbsd
My mind immediately went to Eddie Izzard’s “do you have a flag” bit.
> It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.
Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.
Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production.
But thanks for replying, anyway.
Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.
Nothing in the middle.
There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website.
The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.
Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).
The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.
I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking.
One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it.
There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it.
https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about.
It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.
Kubeadm is a pretty barebones option, which I think needs work to scale (you're on the hook to add all the bells and whistles to create a full k8s environment)
There's validating admission policy coming along (https://kubernetes.io/docs/reference/access-authn-authz/vali...) beta in 1.28, but you still need the policies to apply.
Really for a more "batteries included" kind of cluster, you'd need one of the larger distributions that tries to do more out of the box. Obviously OpenShift is the ultimate example of that (for on-prem) but it massively increases the complexity of the environment.
Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.
Of course not, unless you configure it this way.
> RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security.
What? How comes no one uses service accounts? They're used extensively for many things.
K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.
RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s)
Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common
Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now.
I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)
Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings.
The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS.
Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege.
Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev.
You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.
The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point.
I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references.
for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.
I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box.
I'll stay with swarm for now
I think they were saying that is akin to the state of kubernetes, where everything is too new to really judge it today. Let me make clear my bias before I get to my opinion: I'm a big fan of what kubernetes has accomplished with managing the deployment of applications. But I think comparing current day kubernetes to a restaurant's grand opening is naïve. Kubernetes is amazing at what it does, and for how highly flexible it is. But it is highly flexible because of how advanced it has become, which in turn requires more knowledgeable administrators. You can take off with Docker Swarm because... it's very simple. It doesn't aim to be flexible, it just aims to be the docker-compose of distributed systems. Which is fine if you don't need the flexibility that kubernetes offers anyway.
And while I do appreciate kubernetes for all of its strengths, going back towards the point of this OP, I do believe it's a bit of a security nightmare. For example, the kubernetes operator pattern often just requires this application to run in your cluster with a ClusterRole granting it read and potentially write levels of access to ConfigMaps and Secrets around your cluster. Often without much in the ways of a method of telling the kubernetes API server what namespaces it is allowed to view those objects in. Worth mentioning though that secrets management is something that Docker Swarm also handles fairly poorly, and honestly isn't much of a security regression from the normal "deploy a VM that has all your decrypted secrets sitting on the system" anyway.
I wanted to eat some food so I went to the meat packing factory, hence disappointed I went to a drive through and bought burger.
https://news.ycombinator.com/newsguidelines.html
I've had decent luck in the past with the K3s distribution, which is a bit cut down (but certified) Kubernetes: https://k3s.io/ Others might also mention K0s, MicroK8s or others - there's lots of options there.
It also integrates nicely with Portainer (aside from occasional Traefik ingress weirdness sometimes), which I already use for Swarm and would suggest to anyone that wants a nice web based UI for either of the technologies: https://www.portainer.io/
But even so, I still run Docker Swarm for most of my private stuff as well and it's a breeze. For my needs, it has just the right amount of abstractions: stacks with services and replicas that use networks and can have some storage in the form of volumes or bind mounts. Configuration in the form of environment variables and/or mounted files (or secrets), some deployment constraints and dependencies sometimes, some health checks and restart policies, as well as resource limits.
If I need a mail server, then I just have a container that binds to the ports (even low port numbers) that I need and configure it. If I need a web server, then I can just run Apache/Nginx/Caddy and use more or less 1:1 configuration files that I'd use when setting up either outside of containers, but with the added benefit of being able to refer to other apps by their service names (or aliases, if they have underscores in the names, which sometimes isn't liked).
At a certain scale, it's dead simple to use - no need for PVs and PVCs, no need for Ingress and Service abstractions, or lots and lots of templating that Helm charts would have (although those are nice in other ways). Unfortunately, Swarm isn't awfully good for job prospects, which I have to admit. Then again, one can sort of say the same about Hashicorp Nomad, yet it's also alive and kicking.
I’m honestly curious about Kubernetes and I want to learn more but wow, it really is hard to grok.
What does it mean that k3s is “certified”? Should I care? What’s the tradeoff?
When you mentioned Traefik my eyes glazed over.
Is there a picture of how all this fits together? One not created by a consultant selling their services?
The biggest thing to understand about Kubernetes is that you cannot understand it as a normal piece of software. You don’t directly control it, but rather, define the conditions under which a collection of automations respond to things going on. This is how it self heals. The whole system resembles more of a garden, where you are providing the conditions in which the plant grows, but you are not directly causing the plant to grow.
For example, if you understand a pod, then the next step is understanding how to keep a pod running, how to detect failures, what go do if it does fail, and how to change out one set of pods for the next. This would be a deployment.
You build your understanding by understanding more independent automations. For example, after understanding a deployment, you can look at how horizontal pod autoscalers work. An HPA knows nothing about a deployment and its internal state, yet will inform the deployment how many pods it should be running based on the metrics it sets. The HPA and the deployment scheduler otherwise work independently, and you can indeed write something that functions as an alternate HPA, and Kubernetes will do it.
There’s also an addon for cluster autoscaling. The cluster autoscaling knows nothing of deployments or HPA. It only checks pods that are being requested, but cannot be scheduled due to constraints. It then add nodes so that pods can be scheduled.
These three different processes operate independently, and don’t share state. They each watch for a limited set of signals and take limited set of actions. If any one of these processes goes down, the system can still limp along. Together, they create a much more complex behavior out of much simpler behaviors.
I’m sure it isn’t and I changed the emphasis to make that line more about the slope of the learning curve.
Thanks for this explanation. The inability to grok the internals is really hanging me up. I’ll embrace the simpler “distributions” as a way to understand the higher level constructs.
Does it also make sense to ignore the fact that this is built on Linux’s containers and namespaces?
Based on my very limited understanding Kubernetes is basically a “cloud operating system”. This is why the underlying components like networking and container managers can be swapped out. But I still don’t grok how the subsystems fit together.
The Linux container/namespace is useful in the sense that operational characteristics matter, and many troubleshooting can end up going deeper and deeper in the stack. But as far as understanding how to _design_ things, knowing how the pure abstractions are composed together lets you be able to apply it to many systems, whether it is Kubernetes or something else.
I suggest reading through the first few chapters of Roy Fielding’s PhD dissertation. He is known better for REST, but his dissertation was actually about architectural elements, properties, and constraints, and then applying to hypermedia as an engine of state.
Once you know how to reason through computing architecture, you can pick apart any architecture, framework, or paradigm, and know how to work with it, and when you fight against it. You don’t grok a system by finding out how specific parts fit with each other, but rather, how things are consequences of the architectural elements, properties, and constraints. Then, the specific way things work (and don’t work!) together pops out.
And don’t be afraid to dive into the messiness behind an abstraction. Knowing and being willing to do that can set you apart in your career.
I think that maybe I should have linked directly to the K3s GitHub, which goes into detail about what it is, what the components are, as well as links to more information about testing: https://github.com/k3s-io/k3s/
Kubernetes is a large and complex system, but doing end to end tests aims to ensure that various implementations work correctly, they have pretty in depth docs on all of this, including conformance tests in particular: https://github.com/kubernetes/community/blob/master/contribu...
So, if a distribution passes all of these, then it's likely that it will give you fewer unpleasant surprises. Here's a list: https://www.cncf.io/certification/software-conformance/
> When you mentioned Traefik my eyes glazed over.
Well, you typically do need an ingress controller that will allow exposing services to the outside and direct network traffic.
Traefik is liked by some, less so by others, but generally works okay - except that its documentation (for K3s in particular) isn't as abundant as it would be when using Nginx. For example, if you want to use a custom SSL/TLS certificate (without ACME) for all domains by default, figuring that out might take a while.
That said, if you don't like it, technically you could swap it for something else, however I personally had a cluster hang once while doing that due to some cleanup action getting stuck.
There are plenty of options: https://kubernetes.io/docs/concepts/services-networking/ingr...
> Is there a picture of how all this fits together? One not created by a consultant selling their services?
I actually found their diagrams to be surprisingly sane: https://docs.k3s.io/architecture
However, do be warned that there are many different ways how to build clusters that might have significant differences.
K3s is just one of the Kubernetes distros - that attempt to provide an opinionated view on how to mix and match the components out there to end up with something that will let you use Kubernetes and have things actually work as expected at the end of the day.
A bit like how we have OS distros like Debian, Ubuntu, RHEL and so on.
I brought the problem up with the higher-ups and they were pretty lethargic about it. Moral of the story I guess is if you're gonna choose to operate infrastructure like Kubernetes make sure you have experienced people trained to do so and don't expect your devs to wear another hat.
There’s no way an app developer is going to be able to do their work while also knowing the ins and outs of cluster mgmt.
Ya, and everywhere I've worked, that team is called "devops". The application developers are a different team. Maybe that's not the norm, but I thought it was.
But then that team is only doing operations and not development. Unless they are committers to k8s.
I started back in 2016 building k8s clusters from scratch via terraform and bash scripting. It was ugly awful but worked thru Covid with zero downtime when we switched to hosted solutions from AWS and Google cloud.
What a lot of people miss about larger k8s deployment is that you can create an internal platform team, like a micro-heroku within the company. The company would have to hire people who can create _products_ and also knowing how to work with infra in order to create a platform. It would be this rare combination of product, dev, ops, and focusing on supporting the rest of the engineering team and giving up on the glory of launching something for end users.
In the early days, computers were huge things that were maintained by a team of specialists (and who actually soldered things together). Getting anything changed involved a series of paper forms and waiting for a specialist to do it for you.
Then came the microcomputer revolution and computers became small things that sat on desks and were maintained by the office geek. Changing anything meant talking to him (it was always him) and it would be done by lunchtime, and you owed him a coffee.
Then companies learned that this meant that no two offices had the same configuration, application suite, or even operating system. IT was centralised, and every office issued with the same stuff. Getting anything changed meant filling in forms and waiting for System Administrators to get around to it.
Then Agile and the Cloud happened and while the rest of the company had to carry on filling in forms, the IT Department turned into Engineering and DevOps was born. Every developer learned how to deploy their stuff to the cloud by themselves.
Then organisations worked out that that meant that applications were deployed badly, using a wide variety of tools and technologies, and essential security practices got skipped because no-one was responsible for them. So DevOps was turned back into SysAdmin and the forms reinvented as JIRA tickets.
And so we go on. There are micro-cycles within these ones, like the one about terminals (are you typing on the actual computer, or on a terminal that talks to the actual computer which is somewhere else? The answer depends on which decade you're in).
The ideal solution is somewhere in between (enough central control for it not to get into a mess, but enough decentralisation to prevent bureaucracy and allow for agility in the actual business). But that is a very hard balance to find and keep.
I feel similarly about fullstack. You’re a front end dev that has written a controller and used an ORM before. Often not too inspiring. Backend devs are still needed, whereas full stack devs form excellent glue.
That’s what DevOps is… or, was. I remember when it was new; that was the whole thing: dev and ops should work together. A break from the norm of the time that dev throws it over the wall to ops and never thinks about it again. What a tragedy that it’s been cargo-culted into its current meaning of “devs do ops” (poorly).
Anyway, cheers. In short, I agree with your take.
In my experience, devops folks usually come from the dev side, not the ops side, and their experience and curiosity reflects that. So we let them focus on shipping line-of-business code while making sure what they ship runs in a competently managed environment, as well as handling data centers and all the more traditional infra stuff.
Also, I don't have many resources at my disposal, k8s requires a separate storage cluster. All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. Way too expensive. And then think further, all this has to be updated sometime. I don't have the brain and time resources to do that.
So, if I want k8s, I need to pay for expensive cloud. Or, I can remain with my 1 server with backups and build systems on demand. Yes, I hear the booksmarties scream "but but failsafety". 1 incubator server, specialized systems when something takes off.
Some companies can afford to pay for expensive cloud, more power to them. I can't. I'm busy solving problems at hand, not dealing with additional problems of a system I don't fully understand.
Yes, k8s is a nice comfortable way, if you can afford it. But it's expensive and wasteful of resources. And we do have a climate crisis.
Everyone assumes you have to run multi node.
You can run each of those roles on the same machine, achieving high availability from 3 total. For learning purposes you can run this all in VMs on a single machine.
There are plenty of good reasons not to try kubernetes when it doesn’t fit your use case, but the overhead it imposes is mostly in time and expertise. Claiming it’s more costly in compute resources, and especially contributing climate crisis, is reaching.
Why? Because Kubernetes has some great ideas on self-healing and resiliency.
But if you are small, and can work with a single instance for availability, then probably should stick with systemd. You could also use podman to get a small bit of the kubernetes ideas for working with containers.
The next level of scale wouldn’t be the full k8s but more like k0s, or k3s.
I'm working on a product that uses something like this internally, basically, we provide multiple preview-environments in the same big server, these are managed by systemd.
This is the first time I get to think that this may be a project by itself, ping me if you have any ideas or want to get into a discussion on this topic.
Kubernetes does not require storage cluster. If you need it, you can run it. But it's not required.
Please think about your arguments when comparing kubernetes with any other solution. If you don't need storage cluster with any other solution, then you don't need it with kubernetes.
Kubernetes at its minimum is 3GB RAM and 0.5 CPU for one control plane. That's all you need. Everything extra is not necessary.
People often try to build some kind of super-resilient cluster with kubernetes, which is suitable for huge company with hundreds of servers, while comparing it with one host docker compose. If your task is simple, don't make it more complex than necessary. Kubernetes can be simple or it can be complex, it depends on you.
What you do need for kubernetes is knowledge and experience. I spent several months tinkering with cluster before I dared to deploy something valuable there. Although I used kubeadm, with cloud offering few things are simpler.
That is only half of the truth. If your application needs a database, and which application doesn't, you have to store its data somewhere. So you either need a storage cluster, or you open the can of warms called local volumes and (anti-)affinity rules.
For smaller application deployments, hosting a separate kubernetes cluster is extremely expensive in terms of time spend to debug infrastructure issues. k8s introduces orders of magnitudes of additional complexity to networking, dns, storage, routing, logging, etc. All in the name of making these exact same things easier for the developer. This can be a good separation of work, if you are large enough to afford a dedicated k8s maintenance team. But often it is not.
Regarding storage, try to build stateless apps that only need temporary storage that don’t need to outlive a request (like uploads). For long lived unchanging assets, you can try including in the container (ie. defaults.json, countries.json, samples.json, etc). For assets that need to be fresh, try to have the app generate at startup (eg. Email vírus definitions) or via init container. I think this will cover most use cases.
This won’t work with apps that depend on shared storage like Wordpress, but Wordpress is horrible and I can’t support you running it :-)
So if you think that you need storage cluster either way, go ahead and deploy it. If you think that you're OK with single unreplicated database, just use local volume, it works. At least I've yet encounter any worm with this thing. For me it just worked.
Another option is to use local volumes and replicated database setup. So you can run, say, two database instances in a replicated mode on two servers, but storage will be simple local volume. It provides some reliability and availability benefits without complex storage cluster solutions. I used cloudnative pg operator for this and it worked for me. I guess one can configure ordinary postgres replication without any operator, I just don't have experience with that.
My opinion is that even single-node kubernetes cluster is worth it. That's because I have some experience and won't need time to learn basics. If someone does not have that experience, then I agree. Deploying kubernetes requires learning curve and that time might be spent for other things.
At least it's hard for me to imagine any kubernetes complication that won't arise with, say, docker. At the same time, kubernetes provides lots of polished software line ingress-nginx, cert-manager, cloudnative-pg which could save some time. And, most importantly, using kubernetes from the start, provides very easy option for moving into cloud, for scaling, for zero-downtime deployments. Those things are valuable by almost everyone.
Your control plane and worker node can be the same machine.
Rancher + Rancher Kubernetes Engine made it a lot easier for us to manage as maintaining that is simply "run this docker container" and it deploys all the k8s gubbins for you. And k3s is a lot more lightweight on resources than k8s. Stuff like certificate rotation and kubernetes version upgrades are handled for us, it's reduced to clicking buttons on UI.
Storage can be local, it's just not / wasn't well explained in the documentation. PersistentVolume[Claim]s can use directories on the host, those PVs/PVCs can be shared between many pods and you can mount subdirectories into each pod using `subPath`.
You can get it all running on a laptop using a couple hundred MB of memory, all in all. So, not going to be ideal for running on a Raspberry Pi, but on par with a lightweight linux desktop environment in terms of resource usage.
I hate the complexity sometimes, but then I still end up having the same learning curves with things like SystemD when I use them anyway - every "run a bunch of stuff on my machine and restart it if it breaks" system seems to be rife with edge cases and behaviours which you have to dig deep into the documentation to see - and we don't _want_ to RTFM if we don't expressly _have_ to :-)
^ Although, okay, SystemD doesn't require you to have random HTTPS authenticated endpoints to talk to it and TLS certificates which expire every year and consists of _one_ init process rather than like 7 interdependent docker containers which can restart at any time and rely on container infrastructure to be running and argh. Okay yeah the flashbacks are happening and I agree with everything you've said again.
a foss alternative is kubeztl - it takes kubectl off the networks - no public IP at all - but w/o users needing agents: https://github.com/openziti-test-kitchen/kubeztl/tree/main
disclosure: i am a maintainer and the software overlay in the middle (helps enforce outbound-only, pre-authorized connects only) needs to be managed (self-hosted foss or hosted saas), so there are still trade-offs.
I instead started writing my own product for pull request assignments and direct Slack notifications for those (my personal frustration).
I needed to get away from it all and decompress. It's been refreshing writing code.
So what is growth? The number of developers you have doing stuff. You hire more and more, and every few months you parade some Patagonia vests through your offices to show them. "Observe, more devs! Growth!"
The problems is, the devs all have to do something.
I stopped feeling productive years ago. It's all just never-ending grind, dealing with complexity that has no f--ing business being there. And the younger engineers have no idea. They think this is normal.
This is mostly what we have done. We want to make money. There are barely 10 of us. We don't have time for make believe work that doesn't bring in extra revenue. We used to play in traffic with conversations like "micro services" and "one repo per service" but our investor killed that mode of business and gave us one more chance to do it right.
I recall the original reason for moving away from one big database being something approximating "it won't scale". How well are those various k8s contraptions scaling relative to the "legacy" verticals they purport to replace? How many of us would go back in an instant if given the opportunity?
I think the biggest fear for most on HN is going through ego death over this. The fact that even your "hard" problem is trivially solved with boring tools that were invented before you were born.
To test this, if you hit the K8s API server unauthenticated, do you get a 401 error or a 403 error? If the latter, you're at risk.
Makes it easy to find out what clusters are running what versions of k8s which is interesting, but not a major security issue.
Whilst it usually was just bound to localhost, I did encounter a distribution that, by default, bound it to the container network, meaning that anyone with access to one pod, got to be cluster admin!
Generally speaking though I don't think any distributions do that now.
What is not clear in the blogs is how did they actually accessed the API server without a valid service token. Secondly, they index alot on IPs, that implied these are k8s cluster that are not managed (EKS, GKE etc).
Is my assumption correct?
You can make that mis-configuration with managed distros, which are generally exposed to the Internet by default.
With that said, I agree that secrets management on k8s is pretty bad (unless you use an external service like Hashicorp Vault, a frequent pattern that I run into with k8s). Even with RBAC and secrets in a namespace being inaccessible from other namespaces, the moment someone has permissions to read secrets from a namespace, its essentially game over for you, as noted in this paragraph.
> Another potential attack vector within a Kubernetes cluster is accessing internal services. ... In cases where authentication is required, the credentials can typically be found within the Kubernetes secrets space.
[1] https://docs.k3s.io/advanced#red-hat-enterprise-linux--cento...
trusted zone is used by overlay networks or node internal stuff (or if you manually put your interface into it)
And the range of things you might want to expose... oh, god only knows. Think about such trivial stuff as FTP or iSCSI -- kinda useful, right? -- but Kubernetes rules for port exposure are idiotic, because you cannot expose ranges. And quite a few of popular Internet protocols assume they can allocate whatever port they want and start talking on it. And then you start fighting Kubernetes... and you'll win, but it will be some custom security solution... with the obvious problem of it being a custom security solution.
Now, add to this that Kubernetes will typically mess with your iptables and maybe also arptables, and if your firewall is based on eg. iptables, then you might be out of luck using it with Kubernetes, and again, will end up rolling something "special".
Obviously, you might also want, say, DNS to work inside your cluster, but that's a separate can of worms.
Also, the beancounters don't like on-prem as much anymore due to the upfront cost outlay.
I've been working with a number of major enterprise accounts and they've all been adopting managed K8s offerings for this very reason.
It turns out complex tools for complex tasks require some skilled operators. Yes kubernetes is overkill for your home lab, no it’s not overkill for enterprise-grade AWS orchestration with hundreds of apps and millions of users. News at 11.
If you are in a situation where you want to follow a decoupled/microservices development pattern (ie. split component ownership across dev teams) as well as manage scalability during high load periods, K8s is hard to beat.
K8s is a useful tool, but can be overkill in a number of situations. Just like a hammer, you can't use it to solve for every usecase.
P.S. F-35s use K8s clusters (or might be K3s) to deploy their EWS.
1. https://www.theregister.com/2020/06/03/kubernetes_b_21_bombe...
I've spent my entire career being on-call for web servers and Elastic Beanstalk is not enough. Deployment failures, deployment speed, deployment safety, upgrades, local testing... There is a reason AWS themselves is pushing EKS and not Elastic Beanstalk, and it's not because Kubernetes is trendy, it's because it's incredibly useful.
Kubernetes was built with no program for how to do important things. Every important thing in Kubernetes is post-hoc, poorly bolted-on patch instead of real solution.
Take for instance upgrades. Don't like upgrades? -- Try downgrades? -- Oh, what do I hear, you seem to not like that either... So, how about user management? -- Also no? -- Well, how about backups and recovery? -- What, again?
Kubernetes solved some easy problems quickly with no program for how to solve hard problems. Security is one of those hard problems Kubernetes has no idea what to do about. The reason? -- Kubernetes, on its own, provides very little. It doesn't know how to do networking -- you need a CNI for it. But there's no good interface in Kubernetes to allow the admin to somehow control in general how the cluster is supposed to behave. Maybe some container somewhere will do something tricky, and will punch a hole in whatever firewall you though you've built around it. And you cannot make sure upfront that's not going to happen. There isn't a single way to punch that kind of hole... and the problem is, often time simply to make things work Kubernetes users will punch those holes. Maybe because they don't know how to do things better, or maybe because there isn't really a better way, or the better way is so tedious and convoluted that nobody will really follow it...
It's unreasonable to expect anyone to be able to deal with this kind of complexity. We need to be able to manage complexity, but Kubernetes gives us no tools to do it. It's the same type of problem as Dijkstra described with "goto" -- it's the tools fault. It creates so much complexity that humans cannot cope. Not only humans, computer programs cannot cope -- too many things to check, to many possible problems to protect yourself against.
Upgrades and downgrades work fantastically. Putting "a firewall in front" is trivial - just... put a firewall in front. I don't know what "punch a hole" means, but if you mean "bind a port", yes, there are indeed a lot of ways to do that (on linux, windows, and osx!), and kubernetes provides excellent, organized, standardized and controllable ways to do that.
Kubernetes does not create complexity in my experience. It's the other way around - developers who have never seen the complexity in the world complain about the complexity embodied in the tools. They think "you're not gonna need it", despite an entire industry of operators saying "actually yes, we do need this, and badly".
I've seen developers who've never built a website in their life complain about React, and it sounds exactly the same as this. You might not need these tools, but they're not unneeded. If you can make a simpler and more powerful tool, please, go ahead, you'll be rich and famous, and everyone from the US Navy to telecoms to where I work will celebrate you.
Aaaaaahahhaahhahaha....
.....wait you were serious?
Just one example off the top of my head: ClusterRole vs Role. Two kinds of role. Difference? One applies to a namespace, the other applies to all namespaces, cluster resources, and non-resources. Why not just make "Role" deal with all of them based on metadata or annotations? Because fuck it, that's why. Oh, and permissions cannot include a "deny" section, they can only "allow". Why? Fuck it, that's why.
You can create a secret, but only in one namespace, and there's no way to automatically share it with multiple namespaces. Why? Fuck it, that's why.
K8s is a shitshow of both complexity and a lack of functionality. It's a design-by-committee atrocity that is so bad it created an entire ecosystem to fill in all the bullshit it doesn't do.
What namespaces do allow for, from a security perspective, is for various controls (e.g. RBAC, Network Policy, Pod Security Admission) to be applied to a given application or group of applications (depending on how you've organized your namespaces) in a relatively controlled fashion.
Pods can’t be “upgraded” individually becuase pods are ephemeral.
Downgrades work fine - replace newer binaries with older binaries.
Upgrades work great too - replace older binaries with newer binaries. Single node is the correct context for upgrades because a node is the failure domain in kubernetes (and in every single other Linux based orchestration system). 1.28 even includes an automated way of doing this in a highly available way per host.
CSIs and CNIs are binaries, which can be upgraded and downgraded like with any other system. Yes, if you upgrade or downgrade major versions there may be breaking changes.
I’ve been a sysadmin and a devops engineer and a cloud platform engineer for approaching 20 years. I’ve flagged your reply as I don’t think it’s appropriate, but I wanted to reply to you anyways because I’m sure beyond your anger and rudeness and lack of empathy, you’re intelligent enough to do a bit of reading and double check that you’re not absolutely wrong on all counts.
At some point you’ll be surrounded only by what you think are idiots, and they’ll exclude you. At that point you might investigate if just possibly you’re not by far the smartest person on earth. It’s possible other people know what they’re doing too.
You probably know about resolv.conf, but you might not know about "search domains". A feature that should've never existed, but was added because "why not?". If you don't know what it does, in first approximation, it creates a domain alias. This is sometimes used to save some typing when using DNS for service discovery in larger clusters in a very hierarchical way (when names start to look like a.b.c.d.e.f.g...com, it's tempting to alias them to something shorter).
So, long story short. Ubuntu LTS had a bug (actually a whole bunch of them) in the same script that was responsible for generation of this file (it's a bad idea to have this file to begin with... but hey, who cares, right?). So, one aspect of the bug was that a variable name "DOMAINS" was used instead of its value. And that became a search address in resolv.conf.
Now, this search works by appending the search path to the domain name, if DNS server tells you it cannot find it. I had no idea DOMAINS was possible to register as a valid domain name. I mean, it should've been a TLD to be discoverable globally, right? -- Well, it's still a mystery to me, but, for some reason domains of the form a.b.c.DOMAINS are resolved to some AWS EC2 instance...
And, the punch line: if you screw up CoreDNS configuration somehow, or something else goes wrong with service discovery inside your Kubernetes cluster on unpatched Ubuntu -- there's a good chance your traffic will go to that EC2 instance. Who's on the other side -- I don't know. I didn't send anything important that way and discovered it because something like Spark operator didn't work... but it could've been much worse.
Oh, btw, I believe resolv.conf is used by different CNIs. Maybe Calico and Flanel. I cannot remember atm. why it was used... or maybe it's some kubelet setting.
Anyways. Larger point is... large Kubernetes clusters are virtually impossible to systematically audit. Too many things can do too many things. Too much trickery goes on in all sorts of system components that deal with networking. It's also impossible to track who does what and who owns what because there's no user identity.
---
I believe that, again, industry adopted an absolute horror show of a program / environment not because it was a great tool, but because "organically" developers started to add "features", creating something that's hard to resist, because, superficially, it creates a lot of value with minimal effort -- but it also lays a trap that is not immediately obvious. And those who find themselves trapped, instead of running away, star adding more "features" to fight the trap. But, in this foolhardy attempt, they only make the trap stronger.
The kubelet will merge the hosts resolver config with the cluster config to generate a resolv.conf file for the pods it runs, resulting in the issue you described. This can cause other annoying dns issues. It has nothing to do with CoreDNS. The only way I've found to disable this kubelet behavior is to start the kubelet with `--resolv-conf=/dev/null`
Don't be skeptical. It just sounds like you think you know better, when you actually have no clue.
> resolved to some AWS EC2 instance
How is this a Kubernetes problem, and not an Ubuntu/AWS issue?
(Databases, of course, always like to eat everything because of poorly-optimized statements, but they've never really been good fits for K8s or containers anyway.)
Some might say that this isn't webscale, but Stack Overflow runs one of the most popular tech websites in the world.. on only nine web servers: https://www.datacenterdynamics.com/en/news/stack-overflow-st... and LiChess is similar: https://lichess.org/costs
If you don't care about high-availability, then yeah. One thing I've learned in my career is that delivering reliable enterprise-grade service is anything but trivial. The level of complexity of any modern stance is high enough that there will inevitably be sharp-edged corner cases which aren't simple or cheap to deal with.
The article about stack overflow mentions this little tidbit.
> That application handles around 6,000 requests per second, and around 2 billion requests per month.
To be able to handle loads of 1M requests per second, your going to need more than one server dispensing results. Especially since only so much data can be stored in memory without running into performance issues.
That said, these kinds of tools are an unique use case. Not all programs need them.
K8s is a way for you to drop a large number of such pieces of software onto a small number of such boxes, and keep the configuration and lifecycle of each of those deployments independent.
Kubernetes is not primarily a scalability tool it is an availability tool.
You can find k8s clusters without auth using Censys as they specifically try to access things like pod lists, and from spinning through lists of those they tend to be obvious honeypots (e.g. lots of ports that wouldn't usually go together being exposed), although I have seen the odd cluster that looks a bit more realistic, unlike Aqua I didn't try accessing it directly as that could be dodgy legally :D
If anyone wants to look at the list, Censys handily has a search query for "kubernetes clusters where the pod list is exposed without authentication" https://search.censys.io/search?resource=hosts&q=services.ku...
The entire "CIS Hardening guide" will give you step by step tutorials on how to solve all of this, you just need to bother looking it up.
Look, you need a trained sysadmin anyway, no matter where you deploy. See for instance all the ubuntu machines out there that never have seen the admin update it.
Or all the terraform scripts that simply get copy pasted by orgs.
While I do agree, this is a problem, there are security conscious distributions if kubernetes out there like RKE2 or "rancher government" to make things much more secure.
This is actually my job. If you need someone how to secure your cluster and make sure stuff like this is very unlikely to happen, you can find the info in my profile. But what it comes down to is that if your DEVs have direct cluster access and you have no monitoring and alerting setup, you are kind of operating on the same level as copying and pasting stackoverflow answers and pushing them directly to prod without code review.
rm -rf /
Looks like Linux is no good, back to the drawing board, time to redo the while cloud everybody.