The article is pretty lite, though I agree with most of the points.
However, I think that, especially in the context context of Kubernetes, this part is completely wrong:
> Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.
Containers are not used in Kubernetes or other similar orchestrators because of their support for bundling dependencies - that is a bonus at best.
Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to, so that multiple processes running on the same system can't accidentally affect each other, and so that a process can't accidentally depend on the system it is running on (including things like open ports). These are key properties for a system that seeks to efficiently distribute workloads on a number of computers without having to understand the specifics of what workload it's running.
They are also used because container registries are a ready made secure Linux distribution-agnostic way of retrieving software and referring to it with a unique name. quay.io/python:3.7 will work on Ubuntu, SuSE, RedHat or any other base system, unlike relying on apt/yum/etc.
I think they mean different things to different people. From the problems I have faced with customer-controlled OS installations, the biggest thing that they offer is configuration isolation (or rather independence). I have seen some truly crazy shit done by customer's administrators and even crazier shit done by management software that they install. Taking away that autonomy is huge.
In the context of Kubernetes, what do containers have to do with cgroups? I get that container runtimes typically provide a nicer interface for setting them up, but Kubernetes doesn’t use those interfaces anyway. If it was just about cgroups, they could have just run a regular process in cgroups.
Containers are a construct created by employing cgroups + namespaces together. Namespaces provide isolations in terms of filesystem, network, users, processes, and probably more.
Yes, but those are features of Linux that containers use; containers don’t have exclusive use of them. My original point was:
> Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.
I stand by that. We might just be arguing semantics about whether that unit of code would be called a “container” or not. I’m leaving open the possibility that it would be (say) a WebAssembly module or static binary.
A container can also give your application assurance that the file system looks a certain way. But you are right that you can get pretty far with bundling dependencies and assets and configuring all nontrivial paths with environment variables.
However, the choice of containers for Kubernetes is a pretty fundamental one. It simplifies distribution of deployables to a single workflow also for programming languages that don't offer monolithic deployables, to legacy application not originally intended for Kubernetes, or that don't follow the 12 factor app principles.
After thinking about this for a few minutes, the author might be on the wrong track connecting this with containers.
But I think there's something really there about "environmental linting". I know deep in my bones I need write access to make log files, but I don't know how many times I've debugged systems lacking this permission.
I know the log path won't be known until runtime, I know the port can be specified at runtime, but I think there's a ton of room for improvement around a tool that says - hey, you're making this set of assumptions about your environment, and you should have these health checks or tests or whatever.
I agree with you about this is what containers give, but I think the author is really on to something about the dev tooling and environment warning about what sorts of permissions are needed to, like, work.
> Containers are not used in Kubernetes or other similar orchestrators because of their support for bundling dependencies - that is a bonus at best.
> Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to
Well, sure, you get both. But the point is that needing a post-build step just to get that is accidental complexity. I'm by no means a Java fan, but the way JVM gives jars as a compile target and lets you run them with some measure of isolation is an example of how things could be.
Only some Java/Scala programs can be compiled as a single binary.
It's been a concept that has been around for years but not progressing to the point where it comes close to negating the need for the JVM in a Production environment.
In the meantime containers works today and is significantly more powerful and flexible.
I just don't follow the argument. You can also use Java tooling to wrap up a container image. Why is it accidental complexity that we settled on a language agnostic target that also wraps up a lot of the process isolation metadata?
Java is an outlier here. For most languages the process of dockerizing a codebase involves learning a separate tool (Docker). As someone who knows Docker, I get the temptation to say "who cares, just write a few lines of Dockerfile", but after talking to a bunch of developers about our container-based tool, having to leave their toolchain to build an image is a bigger sticking point than you might think.
But this is begging the question. If the work was more integrated with the compiler, it would still need to be learned. If your compiler of choice spit out an deployable unit akin to an image you'd still need something akin to the Dockerfile to specify exposed ports and volumes and such, no?
The Dockerfile doesn't specify volumes, those are configured at runtime. In the case of Kubernetes, they're configured in the pod spec.
As for ports, EXPOSE in a Dockerfile is basically just documentation/metadata. In practice ports are exposed at runtime (if using docker) or by configuring services (if using Kubernetes), and these are unaffected by whether a port is exposed in the dockerfile.
IMHO this is how it should be -- if I'm running a containerized database or web server, I want to be able to specify the port that the rest of the world sees it as, I don't want the container creator to decide that.
Everything already is running on the same operating system with Kubernetes where it matters (kernel, runc/crun, etc). Containers are a band aid to wrap an app with additional files/libraries/whatever. In Go for instance, I can include all this stuff at compile time (even conditionally!).
I'd love to see Kubernetes be able to schedule executables directly without using containers by way of systemd nspawn or similar. You could have the "container feel" without the complexity of the tool chain required to build/deploy/run/validate containers.
The tool chain to build/deploy/run/validate is the whole point to containers. It's something that unifies the deployment model of applications, unless you want to commit to a single programming language ecosystem. And not every ecosystem has tools to create a monolithic deployment artifact.
Even if you have monolithic deployment artifacts: by the time you have solved all the use cases that the image distribution workflow cover, you have basically reverse-engineered it. If you have already done that, kudos to you, you indeed don't need that workflow. And you probably will find it very easy to containerise those artifacts should you ever need to.
The best formulation I saw is "containers are a solution to the Rube Goldberg Machine development pattern".
Since dependency resolution problems are hard for many popular languages (Node, Python, Ruby), you are not going to enjoy sharing the installed packages between two apps. Same with shared libraries for compiled languages.
Otherwise something like Nix + cgroups-based runner had a chance to suffice.
The executable piece is nice, but not the whole picture. Configuration and, more importantly, isolation of the runtime, are also huge benefits that come with containers.
No, containers give you things that static binaries just don't. How do you specify the maximum allowed memory for a static binary? The ports it opens? The amount of CPU usage? The locations it will access from the host file system?
Also, how will you distribute this static binary? How do you check that the result of a download is the specified version, and the same that others are downloading? How will you specify its name and version?
By the time you have addressed all of these, you will have re-implemented the vast majority of what standard container definitions and registries do.
The orchestrator can still use cgroups for resource constraints and isolation. Or it could use virtualization - it would be an implementation detail. But devs would not have to build a container.
Binary distribution, versioning and checksumming shouldn't need to be coupled to a particular format.
Obviously docker solves a bunch of disparate problems. That's kind of the objection.
> At that point it's not an implementation detail anymore and that's kind of the point.
Not clear what is meant by this. Can you elaborate?
> You don't have to bash kubernetes because you don't specifically need it for your one use case.
It is the platform teams at my employer deploy to, and with good reason. I think we do need it for our use case. But there are ways it could be improved, and the ability to deploy a simple Linux binary feels like one.
It's what we did in C in the 80s and 90s and all the young hipsters chose to ignore. This industry loves going in circles.
In the same vein, I also heard that the new data exchange hotness is grpc. Reading a little into that thing it sounds very much like the good, old... CORBA. So many wheels reinvented every decade, sigh.
Many of the reasons CORBA failed are the reasons that gRPC is succeeding. CORBA tried to solve the wrong problem at the height of the object orientation fad - it tried to provide a standard mechanism for remote “objects” across languages which have different object semantics. It was a doomed idea from the start.
As for the industry going in circles, it only looks that way if you’re not familiar with the technologies in question. The idea that statically linked binaries are an alternative to containers doesn’t make any sense. In fact, it’s not that uncommon for a container to contain nothing but a statically linked binary. People don’t do that for fun, they do it for the additional features and standardization it provides over a standalone binary.
> Distributed objects had nothing to do with object semantics in a PL.
In a way, that was the problem. In practice for CORBA, there was a problematic relationship between its object model and the object models of its host programming languages.
CORBA was supposed to be language independent, but that meant that native language objects and CORBA objects weren't the same thing. There was a semantic gap, analogous to the one between relational databases and object-oriented languages. In the latter case, you can use an ORM-like solution or even just handwritten data access objects to achieve a native language object layer for your data model. With CORBA, that wasn't really feasible - you had to embed CORBA object implementations in your language of choice, and program to that non-native object model. In some ways it's comparable to .NET, but that has the advantage of using a common underlying runtime across all its language implementations. CORBA didn't have that.
You mention the actor model, but what examples are there of successful cross-language actor implementations? The reason for this is exactly what I'm talking about. The practical consequences of a cross-language distributed object model are problematic.
The closest answers I'm aware of share a common runtime - there's Erlang and Elixir both on the Erlang BEAM runtime, and Akka with Java and Scala on the JVM, with variants for C# that I'm not too familiar with. In both cases, the common runtime determines the shared semantics of the object model, so you don't get the same kind of semantic gap.
It's not a coincidence that what's ended up becoming popular are data-only formats like JSON and service-oriented RPC mechanisms like gRPC or even REST - these don't conflict with native language structures in the same way. As I said, many of the reasons CORBA failed are the reasons that gRPC is more successful - the most critical one being that's CORBA's notion of "object" was the wrong level of abstraction for language-independent distributed computing.
Static linking is even worse. Now you traded a blob containing gigabytes of software making it difficult to audit and maintain with a smaller and more opaque binary blob.
Research shows that containers on docker hub, flatpaks etc are full of unmaintained libraries with vulnerabilities in them.
So no, we need to go back to proper dependency management and shared libraries.
You're right that 'containers' are essentially a higher-level pattern but the main underlying building block is arguably kernel namespaces, not just cgroups.
> The fact that you need a special distribution like minikube, Kind, k3s, microk8s, or k0s to run on a single-node instance. (The fact that I named five such distributions is a canary of its own.)
They serve completely different purposes though.
Some are Docker-based designed to be lightweight for testing e.g Kind, others are designed to be scalable to full clusters e.g k3s.
And I think having the choice is a good thing as it proves that Kubernetes is vendor-agnostic.
> The fact that I named five such distributions is a canary of its own
That can be said for programming language implementations. Or programming languages as a concept, too. Or configuration formats. Or even arbitrary products or libraries that overlap and handle the same domain. Another comment mentioned Linux distributions.
My points is: what is your point?
The whole industry is like this (for better or worse). Picking something you don't like and slapping "it has more than one implementation" is hardly an argumentative against it (or any of the other implementations).
> One way to think of Kubernetes is as a distributed framework for control loops. Broadly speaking, control loops create a declarative layer on top of an imperative system.
Finally: a post about Kubernetes that actually understands what it fundamentally is.
> Kubernetes abstracts away the decision of which computer a pod runs on, but reality has a way of poking through. For example, if you want multiple pods to access the same persistent storage volume, whether or not they are running on the same node suddenly becomes your concern again.
I'm not sure I agree that this itself constitutes a leaky abstraction. Kubernetes still abstracts away the decision of which computer a pod runs on, the persistent storage volume is just another declarative constraint on where a pod can be scheduled. It's no different from specifying you need "40" CPU cores, and thus only being able to be scheduled on nodes with >40 cores.
> There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.
I agree - but you can do that with 'containers'. You just create an empty "FROM scratch" container then copy your binary in. The end result is that your OCI image is just a single .tar.gz file containing your binary alongside a JSON document specifying things like the entrypoint and any labels. Just like a shipping container, this plugs into anything that ships things shaped like that.
You get a bunch of nice stuff for free with this (tagged + content addressable remote storage with garbage collection! inter-image layer caching!), even if you're slinging about WASM binaries I'd still package them as OCI images.
> The use of YAML as the primary interface, which is notoriously full of foot-guns.
There's a lot more to say here, and it's more of a legitimate criticism of Kubernetes than most of the "hurr dur k8s complex" critisisms you commonly see. The ecosystem has kind of centered around Helm as a way of templating resources, but it's... horrible. It's all so horrible. A huge step up from ktml or other rubbish from the past, but go's template language isn't fun.
But I'm not sure how it could have gone any differently. JSON is as standard and simple as it gets and is simple to auto-generate, but it's not user friendly. So either the kubernetes getting started guide starts with "first go auto-generate your JSON using some tool we don't distribute", or you provide a more user-friendly way for people to onboard.
YAML is a good middle ground here between giving users the ability to auto-generate resources from other systems (i.e spitting out JSON and shovelling it into a k8s API) and something user-friendly for people to interact with and view using kubectl/k9s/whatever.
Not many? I use k8s extensively and don’t see YAML vs TOML being anything that could make any sort of difference. It’s just a configuration file and it could be markdown or json or whatever and it wouldn’t make that much of a difference because it gets linted anyway. In the end you’re really just discussing the ups and downs of whatever markup language got chosen.
I'm confused, I thought JSON was created because XML wasn't user friendly. I on the other hand see no user friendliness in either YAML, JSON or XML. Just formatted text with or without tabs. Users don't like standards so whatever you choose somebody will complain.
The more I think about it, the more I realize that generic declarative style, despite sounding very promising, might not be a good fit for modern deployment. Mainly due to technology fragmentation.
There's no one true way to deploy an abstract app, each tech stack is fairly unique. Two apps can look exactly the same from the desired state perspective, but have two extremely different deployment processes. They might even have exactly the same tech stack, but different reliability requirements.
Somehow you need to be able to accommodate for those differences in your declarative framework. So you'll pay the abstraction costs (complexity). But you will only reap the benefits if you're constantly switching the components of your architecture. And that typically doesn't happen: by the time you get to a point where you need a deployment framework, your architecture fairly rigid.
Maybe k8s makes a lot of sense if you're Google. But 99.99% of companies are not Google.
I'm not sure what you're trying to say. You list a number of problems that are addressed relatively well within Kubernetes whilst staying somewhat ergonomic; or at least more so than Chef, puppet, ansible, etc.
I'm not sure if you had the misfortune of working with any of the others, but it's extremely frustrating having to untangle them, especially after the 50th variation of it, because it's always done subtly different either because of scripts or something else. Kubernetes and Kustomize remove all these issues. You learn the language (or schema, I guess) once, and your never have to re-learn idiosyncrasies, dialects, variable nomenclatures, and others used between different projects.
Also, here's little room for creativity with templating languages that break tooling (I'm aware of Helm, but I also dislike Helm personally). I also have a feeling this is what people dislike about Kubernetes. It's boring, but it's meant to be.
I agree very much about the accidental complexity of containers. Ignoring the runtime concerns (cgroups, namespaces, networking, etc.), the main problem seems to have been "we can't figure out how to get Python code to production", and the solution was "just ship a disk image of some developer's workstation to production". To do this, they created a new file format ("OCI image format", though not called that at the time), a new protocol for exchanging files between computers ("OCI distribution spec"), a new way to run executables on Linux ("OCI runtime spec"), and various proprietary versions of Make, with a caching mechanism that isn't aware of the actual dependencies that go into your final product. The result is a staggering level of complexity, all to work around the fact that nobody even bothered trying to add a build system to Python.
Like the author, I tend to write software in a language that produces a single statically linked binary, so I don't need any of this distribution stuff. I don't need layers, I don't need a Make-alike, I don't need a build cache, but I still have to go out of my way to wrap the generated binary and push it to a super special server. Imagine a world where we just skipped all of this, and your k8s manifest looks like:
I dunno. I feel like, as an industry, we spent billions of dollars, founded brand new companies, created a new job title ("Devops", don't get me started), all so that we could avoid making Python output a single binary. I'm not sure that random chaos did the right thing, and you're right to be bitter about it.
You /can/ package python applications into a single binary -- just a zip file prepended with a python shebang. You can do much the same with the base python libraries. No one does it, I presume, because the community is against vendoring dependencies that way, the same way they are against static binaries.
> Through Brooks’ accidental-vs-essential lens, a lot of discussion around when to use Kubernetes boils down to the idea that essential complexity can become accidental complexity when you use the wrong tool for the job. The number of states my microwave can enter is essential complexity if I want to heat food, but accidental complexity if I just want a timer. With that in mind ...
I'm twisting my mind trying to grasp this interpretation in Brooks' complexity paradigm. I'm sure Brooks would be interested to learn there exists so called wrong-tools that can reduce essential complexity to accidental :). I think Brooks would put it that the essential complexity is the same and irreducible, but the accidental complexity is increased when the wrong tool is used.
Consumers don't stop and think about what they really need, or what the simplest solution to their problem is. They equate "more features" to "better." I think it was Joel Spolsky who said (paraphrasing) "nothing we did with our software ever created more new revenue than adding features."
> When you run a container on Kubernetes, you don’t specify which computer they run on; you (hopefully) don’t care.
And then you want to deploy something rack aware and you have to deal with `requiredDuringSchedulingIgnoredDuringExecution` in your yaml. Even greater fun ensues if you need some kind of special-purpose hardware.
A Kubernetes based architecture exposes to the developer and the operator a lot of things that superior hardware can abstract almost completely. Things like Mainframes can offer performance and uptime that easily beats pretty large commodity hardware clusters yet we seem to be pushing in this weird direction as an industry while ignoring why the mainframes came to be.
Think of Kubernetes as a "software defined mainframe".
It's an alternative implementation of many of the same concepts. You don't have to go to IBM and purchase a monolithic box with POWER processors in it.
But the thing is that mainframes abstract a lot of the fault tolerance in hardware thus absolving the developer of a lot of the responsibility and the cognitive overhead.
A mainframe in software pitch sounds a bit like that 90s RAM expansion software. Could it possibly work? To a limited degree, perhaps. Would I want it instead of an extra RAM stick? Hell no!
For example, you can get a pair of enormous 100 Tbps core switch/routers for your data centre from Cisco. Upgrading these is terrifying, they cost an incredible amount of money, and if one fails you lose 50% capacity. Conversely, if you run them active-passive, you waste 50% of your capital investment.
Alternatively, you could use a cluster of 1000 VMs on commodity hardware to run your traffic management, with the rules defined purely in software. This can be an order of magnitude cheaper, scale to infinity, and allow smooth upgrades and change rollouts.
The former is what every large enterprise does, the latter is what the public cloud hosting providers do. You can't argue that it's some sort of "fake substitute" when the clouds operate production networks at a scale that far outstrips anything you'd find in any private data centre!
Similarly, many large enterprises use and prefer monolithic mainframes, but in the cloud the preferred option is something like Kubernetes.
For example, with K8s on a typical cloud, it's possible to use "spot" instances to temporarily scale out workloads at night for batch processing to use tens of thousands of CPU cores. No on-prem mainframe can compete with that on any metric. Not absolute throughput, not price, not price/performance.
> With that in mind, I want to talk about the sources of complexity in Kubernetes.
I don't care about the complexities in Kubernetes, I care about the complexities it (cl)aims to be solving. I reread the article a second time to see if I missed something, but the whole article just seems to be talking about the complexities within kubernetes and justifying those.
I still don't understand why every application must be able to scale to infinity and maintain 99.999% reliability. Not everyone is Google and I feel like we'd be a lot better if if the not-Google companies just recognized that instead of layering on accidental complexity in the form of infinite scaling for your service that gets all of 5,000 requests a month.
It's funny, this article mentions Fred Brook's "No Silver Bullet" and then proceeds to tout a solution developed for Google scale problems as the silver bullet to whatever it is Kubernetes is actually solving. Like what does it even mean to be using kubernetes if you only ever have 1 or 2 instances (pods?) running and deploy infrequently (which I suspect is the norm rather than the exception)?
I have been using Kubernetes for my side projects and personal home services for almost 5 years now. I only ever have 1 pod per service and I deploy pretty infrequently.
For me its because how much easier Kubernetes makes my life. I'm using digital ocean's managed kubernetes offering. This means I don't have to worry about the cluster. The OS and updates and all that are managed for me. Same thing with cluster scaling. I just keep adding or removing services and they make sure the cluster is right sized. Basically I just dont ever have to worry about infra.
The next nice thing is the community support. Do I want a load balancer/reverse proxy with auto certificate renewal? It's a simple install command away from running. How about a HA database? MFA? Block storage? It's like magic.
The third thing is that most services are going to have a pretty similar deployment/testing strategy. So when I want to stand up a new one. I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic.
While I dont use this feature, I like the fact that I know I can move my setup to anywhere I want. Move my cluster into my house? on another cloud provider? This would be really easy to lift and shift.
> The OS and updates and all that are managed for me.
I can see this being a nice feature! I feel like the updates have never been a big issue for me in the past though unless it breaks some sort of dependency. In a typical Linux server `sudo apt-get update` has never been a big pain point for me :)
> I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic.
I just looked it up because I was curious, and it looks like setting up an Apache server on Linux is pretty much this same process, as in copy a couple files and tweak a couple settings. Also, I think my biggest question is if you're only using this for home services why can't they all live on one server and deal with it in a traditional style? Why does each service need its own environment as opposed to one environment with multiple services?
I feel like Kubernetes is hoisting all the traditional problems of deployment and infrastructure up to another level. Which is basically the definition of accidental complexity, adding additional complexity where none was warranted.
Once again, I can see how this is very useful if you're Google and need to ensure that if a service goes down it can spin itself back up on an arbitrary machine, but 99% of homebrewed services probably don't need this capability.
Then again, maybe I just enjoy the infra side of things and hacking together a build system and this is just my bias speaking :)
> I still don't understand why every application must be able to scale to infinity and maintain 99.999% reliability. Not everyone is Google and I feel like we'd be a lot better if if the not-Google companies just recognized that instead of layering on accidental complexity in the form of infinite scaling for your service that gets all of 5,000 requests a month.
I agree with this, but some are using Kubernetes (or other orchestrators) just to be able to utilize containers effectively across multiple nodes.
Containers absolutely solve many problems with application packaging, configuration, deployment, healthchecks, resource limits, port allocations, storage persistence, running multiple versions of the same software concurrently and managing more of the deployments as code.
It's just that Kubernetes is the most popular solution for orchestration, which is why many end up getting pigeonholed into using it, especially if they need a managed container platform in the cloud. Finding hosted/managed Nomad or Swarm is an exercise in futility for the most part.
A person by the name of Venkat Subramaniam actually did a pretty nice presentation about accidental and domain (essential) complexity a few years ago: [video] https://www.youtube.com/watch?v=4MEKu2TcEHM
It's a bit humorous and long winded, but personally I rather enjoyed it and it struck a chord with me, especially the bits about how even things like configuration can be a source of frustration.
In regards to Kubernetes, of course, there's also lots of nuance and cases where it is good, as well as those where something simpler (like Nomad/Swarm/Compose) might be a better fit.
Either way, personally I think that containers are pretty good overall, unless you want to ship Vagrant images or full VM images.
Having worked in two of the largest cloud providers there's only one thing I can say which is that wherever I've seen kubernetees been used it has mostly served to make understanding of the application 10 time more complicated ensuring you spend 10 times longer to fix deployment issues
Once you finally get it working is "ok" at least until you have to reconfigure it...
Not like you couldn't have launch a few containers on "restart always" mode... and no, this is not me not understanding autoscaling and the like, IMHO I think that goes to the realm of accidental complexity since the use cases for that are more a corner case than a necesity for 95% of the scenarios where I've seen it being used
79 comments
[ 3.1 ms ] story [ 160 ms ] threadhttp://curtclifton.net/papers/MoseleyMarks06a.pdf
This one inspired our most recent system architecture.
However, I think that, especially in the context context of Kubernetes, this part is completely wrong:
> Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.
Containers are not used in Kubernetes or other similar orchestrators because of their support for bundling dependencies - that is a bonus at best.
Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to, so that multiple processes running on the same system can't accidentally affect each other, and so that a process can't accidentally depend on the system it is running on (including things like open ports). These are key properties for a system that seeks to efficiently distribute workloads on a number of computers without having to understand the specifics of what workload it's running.
They are also used because container registries are a ready made secure Linux distribution-agnostic way of retrieving software and referring to it with a unique name. quay.io/python:3.7 will work on Ubuntu, SuSE, RedHat or any other base system, unlike relying on apt/yum/etc.
> Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.
I stand by that. We might just be arguing semantics about whether that unit of code would be called a “container” or not. I’m leaving open the possibility that it would be (say) a WebAssembly module or static binary.
However, the choice of containers for Kubernetes is a pretty fundamental one. It simplifies distribution of deployables to a single workflow also for programming languages that don't offer monolithic deployables, to legacy application not originally intended for Kubernetes, or that don't follow the 12 factor app principles.
But I think there's something really there about "environmental linting". I know deep in my bones I need write access to make log files, but I don't know how many times I've debugged systems lacking this permission.
I know the log path won't be known until runtime, I know the port can be specified at runtime, but I think there's a ton of room for improvement around a tool that says - hey, you're making this set of assumptions about your environment, and you should have these health checks or tests or whatever.
I agree with you about this is what containers give, but I think the author is really on to something about the dev tooling and environment warning about what sorts of permissions are needed to, like, work.
> Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to
Well, sure, you get both. But the point is that needing a post-build step just to get that is accidental complexity. I'm by no means a Java fan, but the way JVM gives jars as a compile target and lets you run them with some measure of isolation is an example of how things could be.
And I don't understand this idea of using code bundles as the deployment artefact.
They don't specify how the code should be run e.g. Java version, set of SSL certificates in your keystore etc.
Do you really want Kubernetes to have hundreds of options for each language ? Or is it better to just leave that up to the user.
Java programs would need to be compiled as a standard ELF. I think GraalVM can do this.
It's been a concept that has been around for years but not progressing to the point where it comes close to negating the need for the JVM in a Production environment.
In the meantime containers works today and is significantly more powerful and flexible.
As for ports, EXPOSE in a Dockerfile is basically just documentation/metadata. In practice ports are exposed at runtime (if using docker) or by configuring services (if using Kubernetes), and these are unaffected by whether a port is exposed in the dockerfile.
IMHO this is how it should be -- if I'm running a containerized database or web server, I want to be able to specify the port that the rest of the world sees it as, I don't want the container creator to decide that.
What is the point of Kubernetes at all in your situation.
I'd love to see Kubernetes be able to schedule executables directly without using containers by way of systemd nspawn or similar. You could have the "container feel" without the complexity of the tool chain required to build/deploy/run/validate containers.
Even if you have monolithic deployment artifacts: by the time you have solved all the use cases that the image distribution workflow cover, you have basically reverse-engineered it. If you have already done that, kudos to you, you indeed don't need that workflow. And you probably will find it very easy to containerise those artifacts should you ever need to.
Since dependency resolution problems are hard for many popular languages (Node, Python, Ruby), you are not going to enjoy sharing the installed packages between two apps. Same with shared libraries for compiled languages.
Otherwise something like Nix + cgroups-based runner had a chance to suffice.
Also, how will you distribute this static binary? How do you check that the result of a download is the specified version, and the same that others are downloading? How will you specify its name and version?
By the time you have addressed all of these, you will have re-implemented the vast majority of what standard container definitions and registries do.
Binary distribution, versioning and checksumming shouldn't need to be coupled to a particular format.
Obviously docker solves a bunch of disparate problems. That's kind of the objection.
Static binaries are great but they don't solve every problem and they do to have to.
The vast majority of usecases don't need kubernetes, that's fine too.
You don't have to bash kubernetes because you don't specifically need it for your one use case.
There are plenty of other perfectly valid reasons to bash that dogpile.
Not clear what is meant by this. Can you elaborate?
> You don't have to bash kubernetes because you don't specifically need it for your one use case.
It is the platform teams at my employer deploy to, and with good reason. I think we do need it for our use case. But there are ways it could be improved, and the ability to deploy a simple Linux binary feels like one.
In the same vein, I also heard that the new data exchange hotness is grpc. Reading a little into that thing it sounds very much like the good, old... CORBA. So many wheels reinvented every decade, sigh.
As for the industry going in circles, it only looks that way if you’re not familiar with the technologies in question. The idea that statically linked binaries are an alternative to containers doesn’t make any sense. In fact, it’s not that uncommon for a container to contain nothing but a statically linked binary. People don’t do that for fun, they do it for the additional features and standardization it provides over a standalone binary.
In a way, that was the problem. In practice for CORBA, there was a problematic relationship between its object model and the object models of its host programming languages.
CORBA was supposed to be language independent, but that meant that native language objects and CORBA objects weren't the same thing. There was a semantic gap, analogous to the one between relational databases and object-oriented languages. In the latter case, you can use an ORM-like solution or even just handwritten data access objects to achieve a native language object layer for your data model. With CORBA, that wasn't really feasible - you had to embed CORBA object implementations in your language of choice, and program to that non-native object model. In some ways it's comparable to .NET, but that has the advantage of using a common underlying runtime across all its language implementations. CORBA didn't have that.
You mention the actor model, but what examples are there of successful cross-language actor implementations? The reason for this is exactly what I'm talking about. The practical consequences of a cross-language distributed object model are problematic.
The closest answers I'm aware of share a common runtime - there's Erlang and Elixir both on the Erlang BEAM runtime, and Akka with Java and Scala on the JVM, with variants for C# that I'm not too familiar with. In both cases, the common runtime determines the shared semantics of the object model, so you don't get the same kind of semantic gap.
It's not a coincidence that what's ended up becoming popular are data-only formats like JSON and service-oriented RPC mechanisms like gRPC or even REST - these don't conflict with native language structures in the same way. As I said, many of the reasons CORBA failed are the reasons that gRPC is more successful - the most critical one being that's CORBA's notion of "object" was the wrong level of abstraction for language-independent distributed computing.
Research shows that containers on docker hub, flatpaks etc are full of unmaintained libraries with vulnerabilities in them.
So no, we need to go back to proper dependency management and shared libraries.
They serve completely different purposes though.
Some are Docker-based designed to be lightweight for testing e.g Kind, others are designed to be scalable to full clusters e.g k3s.
And I think having the choice is a good thing as it proves that Kubernetes is vendor-agnostic.
Seriously? Linux still can't even agree on tooling to display UI to the user, and its been like 30 years.
Or how many package managers and their supposed replacements are there?
That can be said for programming language implementations. Or programming languages as a concept, too. Or configuration formats. Or even arbitrary products or libraries that overlap and handle the same domain. Another comment mentioned Linux distributions.
My points is: what is your point?
The whole industry is like this (for better or worse). Picking something you don't like and slapping "it has more than one implementation" is hardly an argumentative against it (or any of the other implementations).
Finally: a post about Kubernetes that actually understands what it fundamentally is.
> Kubernetes abstracts away the decision of which computer a pod runs on, but reality has a way of poking through. For example, if you want multiple pods to access the same persistent storage volume, whether or not they are running on the same node suddenly becomes your concern again.
I'm not sure I agree that this itself constitutes a leaky abstraction. Kubernetes still abstracts away the decision of which computer a pod runs on, the persistent storage volume is just another declarative constraint on where a pod can be scheduled. It's no different from specifying you need "40" CPU cores, and thus only being able to be scheduled on nodes with >40 cores.
> There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.
I agree - but you can do that with 'containers'. You just create an empty "FROM scratch" container then copy your binary in. The end result is that your OCI image is just a single .tar.gz file containing your binary alongside a JSON document specifying things like the entrypoint and any labels. Just like a shipping container, this plugs into anything that ships things shaped like that.
You get a bunch of nice stuff for free with this (tagged + content addressable remote storage with garbage collection! inter-image layer caching!), even if you're slinging about WASM binaries I'd still package them as OCI images.
> The use of YAML as the primary interface, which is notoriously full of foot-guns.
There's a lot more to say here, and it's more of a legitimate criticism of Kubernetes than most of the "hurr dur k8s complex" critisisms you commonly see. The ecosystem has kind of centered around Helm as a way of templating resources, but it's... horrible. It's all so horrible. A huge step up from ktml or other rubbish from the past, but go's template language isn't fun.
But I'm not sure how it could have gone any differently. JSON is as standard and simple as it gets and is simple to auto-generate, but it's not user friendly. So either the kubernetes getting started guide starts with "first go auto-generate your JSON using some tool we don't distribute", or you provide a more user-friendly way for people to onboard.
YAML is a good middle ground here between giving users the ability to auto-generate resources from other systems (i.e spitting out JSON and shovelling it into a k8s API) and something user-friendly for people to interact with and view using kubectl/k9s/whatever.
There's no one true way to deploy an abstract app, each tech stack is fairly unique. Two apps can look exactly the same from the desired state perspective, but have two extremely different deployment processes. They might even have exactly the same tech stack, but different reliability requirements.
Somehow you need to be able to accommodate for those differences in your declarative framework. So you'll pay the abstraction costs (complexity). But you will only reap the benefits if you're constantly switching the components of your architecture. And that typically doesn't happen: by the time you get to a point where you need a deployment framework, your architecture fairly rigid.
Maybe k8s makes a lot of sense if you're Google. But 99.99% of companies are not Google.
I'm not sure if you had the misfortune of working with any of the others, but it's extremely frustrating having to untangle them, especially after the 50th variation of it, because it's always done subtly different either because of scripts or something else. Kubernetes and Kustomize remove all these issues. You learn the language (or schema, I guess) once, and your never have to re-learn idiosyncrasies, dialects, variable nomenclatures, and others used between different projects.
Also, here's little room for creativity with templating languages that break tooling (I'm aware of Helm, but I also dislike Helm personally). I also have a feeling this is what people dislike about Kubernetes. It's boring, but it's meant to be.
Like the author, I tend to write software in a language that produces a single statically linked binary, so I don't need any of this distribution stuff. I don't need layers, I don't need a Make-alike, I don't need a build cache, but I still have to go out of my way to wrap the generated binary and push it to a super special server. Imagine a world where we just skipped all of this, and your k8s manifest looks like:
Meanwhile, if you don't want to orchestrate your deployment and just want to run your app: I dunno. I feel like, as an industry, we spent billions of dollars, founded brand new companies, created a new job title ("Devops", don't get me started), all so that we could avoid making Python output a single binary. I'm not sure that random chaos did the right thing, and you're right to be bitter about it.https://bwackninja.github.io/2020/11/25/packaging-python.htm...
I'm twisting my mind trying to grasp this interpretation in Brooks' complexity paradigm. I'm sure Brooks would be interested to learn there exists so called wrong-tools that can reduce essential complexity to accidental :). I think Brooks would put it that the essential complexity is the same and irreducible, but the accidental complexity is increased when the wrong tool is used.
Two dials to set power and clock-timer is enough. Microwaves with digital panel and led display are a perfect example of accidental complexity.
And then you want to deploy something rack aware and you have to deal with `requiredDuringSchedulingIgnoredDuringExecution` in your yaml. Even greater fun ensues if you need some kind of special-purpose hardware.
Or if you have licensing based on "hardware cores" or "servers".
It's an alternative implementation of many of the same concepts. You don't have to go to IBM and purchase a monolithic box with POWER processors in it.
A mainframe in software pitch sounds a bit like that 90s RAM expansion software. Could it possibly work? To a limited degree, perhaps. Would I want it instead of an extra RAM stick? Hell no!
For example, you can get a pair of enormous 100 Tbps core switch/routers for your data centre from Cisco. Upgrading these is terrifying, they cost an incredible amount of money, and if one fails you lose 50% capacity. Conversely, if you run them active-passive, you waste 50% of your capital investment.
Alternatively, you could use a cluster of 1000 VMs on commodity hardware to run your traffic management, with the rules defined purely in software. This can be an order of magnitude cheaper, scale to infinity, and allow smooth upgrades and change rollouts.
The former is what every large enterprise does, the latter is what the public cloud hosting providers do. You can't argue that it's some sort of "fake substitute" when the clouds operate production networks at a scale that far outstrips anything you'd find in any private data centre!
Similarly, many large enterprises use and prefer monolithic mainframes, but in the cloud the preferred option is something like Kubernetes.
For example, with K8s on a typical cloud, it's possible to use "spot" instances to temporarily scale out workloads at night for batch processing to use tens of thousands of CPU cores. No on-prem mainframe can compete with that on any metric. Not absolute throughput, not price, not price/performance.
I don't care about the complexities in Kubernetes, I care about the complexities it (cl)aims to be solving. I reread the article a second time to see if I missed something, but the whole article just seems to be talking about the complexities within kubernetes and justifying those.
I still don't understand why every application must be able to scale to infinity and maintain 99.999% reliability. Not everyone is Google and I feel like we'd be a lot better if if the not-Google companies just recognized that instead of layering on accidental complexity in the form of infinite scaling for your service that gets all of 5,000 requests a month.
It's funny, this article mentions Fred Brook's "No Silver Bullet" and then proceeds to tout a solution developed for Google scale problems as the silver bullet to whatever it is Kubernetes is actually solving. Like what does it even mean to be using kubernetes if you only ever have 1 or 2 instances (pods?) running and deploy infrequently (which I suspect is the norm rather than the exception)?
For me its because how much easier Kubernetes makes my life. I'm using digital ocean's managed kubernetes offering. This means I don't have to worry about the cluster. The OS and updates and all that are managed for me. Same thing with cluster scaling. I just keep adding or removing services and they make sure the cluster is right sized. Basically I just dont ever have to worry about infra.
The next nice thing is the community support. Do I want a load balancer/reverse proxy with auto certificate renewal? It's a simple install command away from running. How about a HA database? MFA? Block storage? It's like magic.
The third thing is that most services are going to have a pretty similar deployment/testing strategy. So when I want to stand up a new one. I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic.
While I dont use this feature, I like the fact that I know I can move my setup to anywhere I want. Move my cluster into my house? on another cloud provider? This would be really easy to lift and shift.
I can see this being a nice feature! I feel like the updates have never been a big issue for me in the past though unless it breaks some sort of dependency. In a typical Linux server `sudo apt-get update` has never been a big pain point for me :)
> I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic.
I just looked it up because I was curious, and it looks like setting up an Apache server on Linux is pretty much this same process, as in copy a couple files and tweak a couple settings. Also, I think my biggest question is if you're only using this for home services why can't they all live on one server and deal with it in a traditional style? Why does each service need its own environment as opposed to one environment with multiple services?
I feel like Kubernetes is hoisting all the traditional problems of deployment and infrastructure up to another level. Which is basically the definition of accidental complexity, adding additional complexity where none was warranted.
Once again, I can see how this is very useful if you're Google and need to ensure that if a service goes down it can spin itself back up on an arbitrary machine, but 99% of homebrewed services probably don't need this capability.
Then again, maybe I just enjoy the infra side of things and hacking together a build system and this is just my bias speaking :)
Then again, dealing with that is also something Kubernetes can help with.
I agree with this, but some are using Kubernetes (or other orchestrators) just to be able to utilize containers effectively across multiple nodes.
Containers absolutely solve many problems with application packaging, configuration, deployment, healthchecks, resource limits, port allocations, storage persistence, running multiple versions of the same software concurrently and managing more of the deployments as code.
I've actually written about my personal experiences and why containers helped me before, "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao...
It's just that Kubernetes is the most popular solution for orchestration, which is why many end up getting pigeonholed into using it, especially if they need a managed container platform in the cloud. Finding hosted/managed Nomad or Swarm is an exercise in futility for the most part.
It's a bit humorous and long winded, but personally I rather enjoyed it and it struck a chord with me, especially the bits about how even things like configuration can be a source of frustration.
In regards to Kubernetes, of course, there's also lots of nuance and cases where it is good, as well as those where something simpler (like Nomad/Swarm/Compose) might be a better fit.
Either way, personally I think that containers are pretty good overall, unless you want to ship Vagrant images or full VM images.
Once you finally get it working is "ok" at least until you have to reconfigure it...
Not like you couldn't have launch a few containers on "restart always" mode... and no, this is not me not understanding autoscaling and the like, IMHO I think that goes to the realm of accidental complexity since the use cases for that are more a corner case than a necesity for 95% of the scenarios where I've seen it being used