My guess Linux started getting requests rom various orgs for a while, so in true Linux fashion, we got a a few different container type methods years later.
I still think Jails are the best of the bunch, but they can be a bit hard to setup. Once setup, Jails works great.
Containers happened because running an ad network and search engine means serving a lot of traffic for as little cost as possible, and part of keeping the cost down is bin packing workloads onto homogeneous hardware as efficiently as possible.
Fascinating documentary on Kubernetes for those who have 50 minutes. Gives more background to the "Container Wars". The filmmakers also have documentaries on the history of Python, Argo, etc.
Some highlights:
- How far behind Kubernetes was at the time of launch. Docker Swarm was significantly more simple to use, and Apache Mesos scheduler could already handle 10,000 nodes (and was being used by Netflix).
- RedHat's early contributions were key, despite having the semi-competing project of OpenShift.
- The decision to Open Source K8S came down to one meeting brief meeting at Google. Many of the senior engineers attended remotely from Seattle, not bothering to fly out because they thought their request to go OS was going to get shutdown.
- Brief part at the end where Kelsey Hightower talks about what he thinks might come after Kubernetes. He mentions, and I thought this was very interesting ... Serverless making a return. It really seemed like Serverless would be "the thing" in 2016-2017 but containers were too powerful. Maybe now with KNative or some future fusing of Container Orchestration + K8S?
Containers (meaning Docker) happened because CGroups and namespaces were arcane and required lots of specialized knowledge to create what most of us can intuitively understand as a "sandbox".
Cgroups and namespaces were added to Linux in an attempt to add security to a design (UNIX) which has a fundamentally poor approach to security (shared global namespace, users, etc.).
It's really not going all that well, and I hope something like SEL4 can replace Linux for cloud server workloads eventually.
Most applications use almost none of the Linux kernel's features. We could have very secure, high performance web servers, which get capabilities to the network stack as initial arguments, and don't have access to anything more.
Drivers for virtual devices are simple, we don't need Linux's vast driver support for cloud VMs. We essentially need a virtual ethernet device driver for SEL4, a network stack that runs on SEL4, and a simple init process that loads the network stack with capabilities for the network device, and loads the application with a capability to the network stack.
Make building an image for that as easy as compiling a binary, and you could eliminate maybe 10s of millions of lines of complexity from the deployment of most server applications. No Linux, no docker.
Because SEL4 is actually well designed, you can run a sub kernel as a process on SEL4 relatively easily. Tada, now you can get rid of K8s too.
The story I heard was that containers let you use less memory and better share the kernel and CPU compared to Virtual Machines, such that you could run more applications on the same servers. This translates into direct cost savings, which is why large companies with large server farms were willing to pay their engineers to develop the technology and transition to the technology.
In terms of security, I think even more secure than SEL4 or containers or VMs would be having a separate physical server for each application and not sharing CPUs or memory at all. Then you have a security boundary between applications that is based in physics.
Of course, that is too expensive for most business use cases, which is why people do not use it. I think using SEL4 will run into the same problem - you will get worse utilization out of the server compared to containers, so it is more expensive for business use cases and not attractive. If we want something to replace containers that thing would have to be both cheaper and more secure. And I'm not sure what that would be
it happened because the story of dependencies (system & application) was terrible. the ability to run the app on different distribution/kernel/compiler/etc was hard. there were different solutions like vagrant, but they were heavy and the DX wasn't there
For me the main reason to use containers is "one-line install any linux distro userspace". So much simpler than installing a dozen VirtualBox boxes to test $APP on various versions of ubuntu, debian, nixos, arch, fedora, suse, centos etc.
I love this sentence about DevOps "Somehow it seems easier for people to relate to technology than culture, and the technology started working against the culture."
My headcanon is that Docker exists because Python packaging and dependency management was so bad that dotCloud had no choice but to invent some porcelain on top of Linux containers, just to provide a pleasant experience for deploying Python apps.
> I was always surprised someone didn't invent a tool for ftping to your container and updating the PHP
We thought of it, and were thankful that it was not obvious to our bosses, because lord forbid they would make it standard process and we would be right back where we started, with long lived images and filesystem changes, and hacks, and managing containers like pets.
The author suggests that Docker doesnt help development and that devs just spin up databases, but I have to disagree with that and Im pretty sure i am not the only one.
All my projects (primarily web apps) are using docker compose which configures multiple containers (php/python/node runtime, nginx server, database, scheduler, etc) and run as a dev environment on my machine. The source code is mounted as a volume. This same compose file is then also used for the deployment to the production server (with minor changes that remove debug settings for example).
This approach has worked well for me as a solo dev creating web apps for my clients.
It has also enabled extreme flexibility in the stacks that I use, I can switch dev environments easily and quickly.
I loved the assertion that AI ate up all the budget and that K8s is now "boring" technology. That's fine because it was getting pretty annoying with all the clone competitors for practically everything that were popping up every month!
Do you use K8s? No! That's old! I use Thrumba! It's just a clone of K8s by some startup because people figured out that the easiest way to make gobs of money is/was to build platform products and then get people to use them.
Containers happened because nobody can be bothered to build an entire application into a single distributable executable anymore - heck even the tooling barely exists anymore. But instead of solving problems like dependency management and linking, today's engineers simply build piles of abstraction into the problem space until the thing you want to do more than anything (i.e. execute an application) becomes a single call.
Of course you now need to build and maintain those abstract towers, so more jobs for everybody!
by "today's engineers", do you mean "2001 engineers"?
That's when sbuild[0], a tool to build deb packages in containers, was created. It was pretty innovative in that it started from clean container every time, and thus would build deb in a reliable way even if user's machine had some funky dependencies installed.
(Note that was schroot containers, docker did not exist back then)
My take: containers forced devepopers to declare various aspects of the application in a standardized, opinioated way:
- Persistant state? Must declare a volume.
- IO with external services? Must declare the ports (and maybe addresses).
- Configurable parameters? Must declare some env variables.
- Trasitive dependecies? Must declare them, but using a mechanism of your choosing (e.g. via the package manager of your base image distro).
Separation of state (as in persistency) and application (as in binaries, assets) makes updates easy. Backups also.
Having most all IO visible and explicit simplifies operation and integration.
And a single, (too?!?) simple config mechanism increases reusability, by enabling e.g. lightweight tailoring of generic application service containers (such as mariadb).
Together this bunch of forced, yet leaky abstractions is just good enough to foster immense reuse & composability on to a plethora of applications, all while allowing to treat them almost entirely like blackboxes. IMHO that is why OCI containers became this big, compared to other virtualization and (application-) cuntainer technologies.
Yeah. I remember when "okay, cool, how do we deploy, run, maintain the thing we (our leadership) just bought from you" was several weeks of meetings with the vendor and discussing awful "human-readable" documentation (if it existed).
Those meetings still happen sometimes, but "cloud" *aaS and containers really put a dent in that sort of thing.
Why it happened is not nearly as important as what it unveiled: that versioned immutable systems are the most powerful system design concept in history. Most people have not yet grasped what an insanely powerful concept it is. At some point in the future, maybe 50-100 years from now, someone will look back and say "holy shit; deploying web apps are where this concept came from?" I hope in my lifetime that people get it and start applying it to other system designs/components.
in some situations, the cloud is a no-go and the lead time to install new hardware is quite lengthy. in those case vm's or containers allow for rapid change at the software/OS level while the meat space still moves like a glacier.
> Application composition from open source components became the dominant way of constructing applications over the last decade.
I'm just as interested in why this ^ happened. I imagine it's pretty unique to software? I don't hear of car companies publishing component designs free for competitors to use, or pharmaceuticals freely waiving the IP in their patents or processes. Certainly not as "the dominant way of" doing business.
I wonder if LLM coding assistants had come about earlier, whether this would have been as prevalent. Companies might have been more inclined to create more of their own tooling from scratch since LLMs make it cheap (in theory). Individuals might have been less inclined to work on open source as hobbies because LLMs make it less personal. Companies might be less inclined to adopt open-source LLM-managed libraries because it's too chaotic.
containers happened because the original execution isolation environment(the process) was considered a lost cause, Processes shared too much with each other so additional isolation primitives had to be added, but they had to be sort of tacked on to the side because more important than security or correctness is backwards compatibility. so now containers are considered a different thing than processes when really they are process with these additional isolation primitives enabled.
60 comments
[ 2.3 ms ] story [ 71.4 ms ] thread>hosting provider's ... desire to establish a clean, clear-cut separation between their own services and those of their customers
https://en.wikipedia.org/wiki/FreeBSD_jail
My guess Linux started getting requests rom various orgs for a while, so in true Linux fashion, we got a a few different container type methods years later.
I still think Jails are the best of the bunch, but they can be a bit hard to setup. Once setup, Jails works great.
So here we are :)
https://en.wikipedia.org/wiki/Cgroups
(arguably FreeBSD jails and various mainframe operating systems preceded Linux containers but not by that name)
Some highlights:
- How far behind Kubernetes was at the time of launch. Docker Swarm was significantly more simple to use, and Apache Mesos scheduler could already handle 10,000 nodes (and was being used by Netflix).
- RedHat's early contributions were key, despite having the semi-competing project of OpenShift.
- The decision to Open Source K8S came down to one meeting brief meeting at Google. Many of the senior engineers attended remotely from Seattle, not bothering to fly out because they thought their request to go OS was going to get shutdown.
- Brief part at the end where Kelsey Hightower talks about what he thinks might come after Kubernetes. He mentions, and I thought this was very interesting ... Serverless making a return. It really seemed like Serverless would be "the thing" in 2016-2017 but containers were too powerful. Maybe now with KNative or some future fusing of Container Orchestration + K8S?
[1] - https://youtu.be/BE77h7dmoQU
Cgroups and namespaces were added to Linux in an attempt to add security to a design (UNIX) which has a fundamentally poor approach to security (shared global namespace, users, etc.).
It's really not going all that well, and I hope something like SEL4 can replace Linux for cloud server workloads eventually. Most applications use almost none of the Linux kernel's features. We could have very secure, high performance web servers, which get capabilities to the network stack as initial arguments, and don't have access to anything more.
Drivers for virtual devices are simple, we don't need Linux's vast driver support for cloud VMs. We essentially need a virtual ethernet device driver for SEL4, a network stack that runs on SEL4, and a simple init process that loads the network stack with capabilities for the network device, and loads the application with a capability to the network stack. Make building an image for that as easy as compiling a binary, and you could eliminate maybe 10s of millions of lines of complexity from the deployment of most server applications. No Linux, no docker.
Because SEL4 is actually well designed, you can run a sub kernel as a process on SEL4 relatively easily. Tada, now you can get rid of K8s too.
In terms of security, I think even more secure than SEL4 or containers or VMs would be having a separate physical server for each application and not sharing CPUs or memory at all. Then you have a security boundary between applications that is based in physics.
Of course, that is too expensive for most business use cases, which is why people do not use it. I think using SEL4 will run into the same problem - you will get worse utilization out of the server compared to containers, so it is more expensive for business use cases and not attractive. If we want something to replace containers that thing would have to be both cheaper and more secure. And I'm not sure what that would be
So much this. keep it simple, stupid (muah)
We thought of it, and were thankful that it was not obvious to our bosses, because lord forbid they would make it standard process and we would be right back where we started, with long lived images and filesystem changes, and hacks, and managing containers like pets.
All my projects (primarily web apps) are using docker compose which configures multiple containers (php/python/node runtime, nginx server, database, scheduler, etc) and run as a dev environment on my machine. The source code is mounted as a volume. This same compose file is then also used for the deployment to the production server (with minor changes that remove debug settings for example).
This approach has worked well for me as a solo dev creating web apps for my clients.
It has also enabled extreme flexibility in the stacks that I use, I can switch dev environments easily and quickly.
Do you use K8s? No! That's old! I use Thrumba! It's just a clone of K8s by some startup because people figured out that the easiest way to make gobs of money is/was to build platform products and then get people to use them.
Of course you now need to build and maintain those abstract towers, so more jobs for everybody!
That's when sbuild[0], a tool to build deb packages in containers, was created. It was pretty innovative in that it started from clean container every time, and thus would build deb in a reliable way even if user's machine had some funky dependencies installed.
(Note that was schroot containers, docker did not exist back then)
[0] https://metadata.ftp-master.debian.org/changelogs//main/s/sb...
- Persistant state? Must declare a volume. - IO with external services? Must declare the ports (and maybe addresses). - Configurable parameters? Must declare some env variables. - Trasitive dependecies? Must declare them, but using a mechanism of your choosing (e.g. via the package manager of your base image distro).
Separation of state (as in persistency) and application (as in binaries, assets) makes updates easy. Backups also.
Having most all IO visible and explicit simplifies operation and integration.
And a single, (too?!?) simple config mechanism increases reusability, by enabling e.g. lightweight tailoring of generic application service containers (such as mariadb).
Together this bunch of forced, yet leaky abstractions is just good enough to foster immense reuse & composability on to a plethora of applications, all while allowing to treat them almost entirely like blackboxes. IMHO that is why OCI containers became this big, compared to other virtualization and (application-) cuntainer technologies.
Those meetings still happen sometimes, but "cloud" *aaS and containers really put a dent in that sort of thing.
sounds minor, but it is a Big Deal for some
No FTP needed, you can just mount the application directory.
I'm just as interested in why this ^ happened. I imagine it's pretty unique to software? I don't hear of car companies publishing component designs free for competitors to use, or pharmaceuticals freely waiving the IP in their patents or processes. Certainly not as "the dominant way of" doing business.
I wonder if LLM coding assistants had come about earlier, whether this would have been as prevalent. Companies might have been more inclined to create more of their own tooling from scratch since LLMs make it cheap (in theory). Individuals might have been less inclined to work on open source as hobbies because LLMs make it less personal. Companies might be less inclined to adopt open-source LLM-managed libraries because it's too chaotic.