> One of Podman’s greatest advantages is its complete CLI compatibility with Docker. In fact, when building Podman, Docker users can adapt without any significant changes. For example, you can use the alias command to create a docker alias for Podman:
`alias docker=podman`
Wow that's a great adoption hack; just alias another program as another
I like the daemonless architecture a lot, but until there's a quick and painless way to install it on OSX and Windows developer machines, you're going to see very limited uptake.
I find it somewhat surprising that we haven't seen more adoption of the redhat docker (lowercase d) replacements after Docker (uppercase d) started pushing some of their monetization efforts.
The conclusion that I have to reach is that there are more docker users on Macbooks than I realized.
I was just thinking the same thing, and was going to ask if there was anything like Docker Desktop, something to make it simple to use on Windows/MacOS!
I actually like the idea of Docker as a better systemd (or rather, the interface is better). No bespoke file format, programmable API, no need to google for the right journalctl switches, and then of course the advantages of containers and images over processes and system packages. I’m not suggesting everything should be a container nor that docker is the ideal implementation, but it certainly points in the right direction.
But systemd has all of those things. Everything in systemd can be orchestrated with dbus. In fact that’s all systemctl really is — a dbus client. And the file format is ini. And shell style environment files if you want.
And systemd actually comes with two image launching systems systemd-nspawn and systemd-portable. And then with systemd-machined you can add software that needs virtualization too.
The interface to journald is more complicated than it should be but it’s also really powerful — docker logs doesn’t hold a candle to the kind of filtering it can do.
Overall systemd is a stupidly powerful and featureful supervisor compared to Docker. Just the dependency management alone should demonstrate that. Then you get mounts, swap, socket activation, more powerful restarting policy and the whole suite of isolation and security features.
> The interface to journald is more complicated than it should be but it’s also really powerful
I thought the point of logging to stdout (i.e. docker logs) is that you just take that output and dump it to another server for processing and filtering.
journald/journalctl seem to be a solution a few decades late to the party. For a single user machine or a single app prod environment, I would take a plaintext log any day of the week. At least I can remember how to grep the damn thing. And then when you get to a distributed system, what's the point of journalctl? You would hopefully have all of that logging aggregated together in one place with a much nicer web interface for it all.
Exactly this. I just want a stream of logs that I can process with ordinary unix tools without jumping through hoops. I don’t need any extra power for its own sake.
Not a huge deal, but I have to google it every few months because I don’t use it very often.
Note also that if I just want the log stream, I have to pipe through less to get the full log messages. There’s also a flag for it, but I can’t remember my workaround is easier than digging through man pages.
Again, no big deal, just friction. Like everything in the systemd ecosystem—everything is manageable, but it’s tedious.
Last time i did look into docker it couldn't even tell if a service is up or still starting and there were all kinds of weird hacks around that like "is something listening on the configured port". Did that improve?
Systemd struggles with this as well. While a service can easily tell you if it's "up", sometimes the only way to determine if it's "available" is to check. Keep in mind that the sentinel for any given service will be different. If you think about it, how could any external monitoring service determine when a service under it's purview is available without actually checking?
Maybe it already has something like this, but I feel like systemd could provide an API endpoint for applications to send a simple status when they are "ready" - at which point it would be up to the developers to provide that at the right time.
From the link: "systemd-notify may be called by daemon scripts to notify the init system about status changes. It can be used to send arbitrary information, encoded in an environment-block-like list of strings. Most importantly, it can be used for start-up completion notification."
> I actually like the idea of Docker as a better systemd (or rather, the interface is better). No bespoke file format, programmable API, no need to google for the right journalctl switches, and then of course the advantages of containers and images over processes and system packages. I’m not suggesting everything should be a container nor that docker is the ideal implementation, but it certainly points in the right direction.
Docker was created to give an extra life for legacy applications depending on outdated packages, the concept of gluing all dependencies withing a compressed rootfs and shipping that, with low effort.
Docker became popular because of that, but seems that did more harm than good for the ecosystem in terms of security, that why podman might have a great future. The docker interface is bad and comparing that with systemd is quite a stretch. ;)
Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.
Aren't containers just processes with namespaces and cgroups?
> The docker interface is bad and comparing that with systemd is quite a stretch. ;)
I don't know, systemctl is always a pain to interact with. The Docker CLI is pretty intuitive--you have different resources and different verbs for interacting with those resources. This includes logs. No need to use a separate (and confusing) tool nor dig through man pages. I'm of the philosophy that it's better for a tool to be intuitive than obscure-but-has-manpages.
> Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.
I guess I meant that INI isn't a standard file format--different parsers behave differently, and structured data is often coded in strings in some bespoke format.
> Aren't containers just processes with namespaces and cgroups?
My mind isn't made up that containers are the ideal process unit, but I do like some things about them (and the container ecosystem more generally). That they come with their dependencies bundled is pretty nice, but I think the toolchain needs to improve to mitigate security concerns and so on. Something like Nix would be helpful, but Nix also has its own problems. Again, it's directionally correct. In any case, users don't need to be managing their own namespaces and cgroups.
How are you using Docker that doesn't need a bespoke file format? You're building from a Dockerfile and either scripting around the docker command or running docker-compose.yml
Like I said, I don’t actually want docker or docker compose, but they are directionally correct. Notably YAML over INI is a nice improvement. Also, that the docker daemon has a real API and not just dbus.
Well, Docker has an API and its own file format (the Dockerfile), I constantly have to look up Docker switches as it's not a tool I use every day and it's not at all intuitive, and I'm still unclear why people love this layered/stack image approach given how unbelievably slow it makes everything. Apps that would once have been a 50mb tarball are now routinely a gigabyte of downloads, perhaps because whatever random version of Ubuntu I downloaded previously is now slightly different? Who knows.
There is, frankly, a lot of appeal to me in a simple INI file that starts up something I unzipped.
Any docker Podman corner cases that people have run into? I like the idea of rootless containers but I can do that ether easily by adding one line to my dockerfile or adding a user flag when I run a docker container. What other advantages am I getting?
The biggest issue lies not in the few lines in /etc/passwd or /etc/group, but rather the highly privileged process with a large attack surface that is the Docker daemon.
Generally, it's been pretty easy to transition for me, but I used to rely on things like the automatic nginx image that you need to mount the docker socket into so it can route traffic to containers when they come up. Since there's no daemon, there wasn't a way to do that. I think people are working on it, but I haven't tried recently.
Work also uses a special docker RUN command that can use the host's ssh keyring to, eg. Install something from a private github repo. Doesn't seem to exist in podman.
I use that nginx image too, and it's the one thing that keeps me using docker for some small single-node setups. For basically everything else, Podman is a better experience. Now that I've discovered Podman can run K8s Pods (and now Deployments) I use it all the time. I don't know how I lived without it!
Not sure if you count it as a corner case, but podman lacks a docker-compose equivalent - last I checked, a few months ago, there was only a community port project that still needed a lot of work.
I've also personally struggled a lot with this at $WORK, but some of the new features in RHEL 7.8+ (specifically GA'ing of the fuse-overlayfs graph driver) have actually alleviated a ton of the issues I've seen previously. If you haven't tried that I'd certainly recommend it, YMMV
I just installed my new $WORK docker-compose using podman and podman-compose, and it worked out of the box. To be fair, I failed at it one time before at previous job... So... Coin flip?
I was just about to complain about another thing to learn. Then I saw you can do "$ alias docker=podman". Just want to acknowledge the importance of that work. Making things compatible is both boring and a pain, but it's a door opener for people like me who refuse to learn your new API because I know a decent one already.
One thing I want to point out though for anybody not familiar with the differences between podman and docker, for the most part alias docker=podman will "just work" except for these situations:
1. docker-compose. podman-compose attempts to cover this but I've heard it's not quite there yet
2. Mounting the docker socket into the container. Podman is daemonless which means that won't work. There is work going on right now to allow Podman to be driven in a similar way if needed, but I recently tried to set it up and hit a bug[1]. CRI-O brings a daemon, and it is used extensively in Kubernetes and OpenShift, but not so much outside of that.
Unless you're optimizing for stagnation, this really is a terrible approach to reaping the benefits of innovation. If your prereq to trying something better is that it needs to mirror something that's "decent already," you'll spend most of your life stuck on the same plateau.
Yeah, you're right. I should make the time to learn more stuff and never make excuses for why I don't.
On the other hand, I don't think you fully understand where I'm coming from. "Optimizing for stagnation", "stuck on the same plateau". Idk, I just want to run a container. I'm totally fine being on the same Docker plateau if something like Podman makes me re-learn what I already know how to do.
Unlike Git, which, from the start, was different and BETTER than SVN, can you really make the case that Podman would be BETTER if it didn't try to follow Docker conventions? If not, what are we talking about here?
History is full of technologies that are superior to existing options on one metric or another and yet failed to take over due to switching costs. Attacking someone as “optimizing for stagnation” is a really bizarre way to view their reaction to a product offering a smooth on boarding process.
So far, it's just a less functional docker and didn't bring much to the table.
I don't feel the need for daemonless as docker has been working fine.
I've swapped back to docker on Fedora and CentOS by forcibly installing docker on those unsupported platforms because podman-compose doesn't work, containers just didn't work like expected in several situations (forgot detail) and portainer wouldn't work against it.
Either they should have sped up the podman development or shouldn't have deprecated docker at this point. Maybe RedHat just didn't like a third party taking the stewardship in the container business.
At my company, we run our CI/CD (Jenkins) using the Docker-in-Docker paradigm to facilitate easy maintainability of the CI itself and allow us to run containerized builds.
When we shifted to RHEL 8, we attempted to move this over to Podman and it went miserably (this was back in November 2019). The main reason being is that podman-in-podman doesn't work and had bugs (at least back in Nov 2019). Maybe it fixed now but this was our experience. We ended up doing quite a bit of analysis on podman only to conclude it's simply not there yet relative to docker (ecosystem and ergonomics).
There are quite a few corner cases that docker quite simply supports out of the box beautifully that podman doesn't support or just has bugs.
I like the what the project is trying to solve by being daemonless, but this is not as simple as a drop in replacement for docker that RedHat markets it as (alias docker=podman).
We ended up sticking to docker professionally and personally, I am still using docker over podman. The ecosystem and ergonomics are just far too nice to give up over podman.
Maybe you are just too far into docker. I noticed that a lot of default workflows (needlessly) depended on docker running with privileges. One big reason for that seem to be Mac users that only know docker from inside a VM. However, if you think about what you're really needing for CI you will easily see that docker-in-docker gains you nothing. You can as well use plain docker (or podman). The same holds for privileges. No CI operation should need privileges, if only for the reason that it should never alter the CI system itself.
I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.
Why do you say that docker-in-docker buys him nothing? It's not obvious at all and you go into no detail whatsoever to back up your opinion.
In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.
>> beastly 5G docker image
> my beastly 12GB image that even includes Matlab wants a word with you
Perhaps in the next 10 years we will be rediscovering packages. :P
If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
> If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
True, that.
To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
Well, to be fair, it is packages - I'm just using Docker (for this section of our stack) as a different sort of VM, essentially. It runs a service manager and a VNC X session, for chrissakes ;)
Our images are 35GB, and I've spent much of the last two weeks breaking up files so we don't hit the 8GB per file limit, and my next week will be trying to avoid hitting the per-layer limit.
It is because to build the image using docker requires the docker client to talk with a dockerd daemon, so one has to configure the client to access the dockerd which allow untrusted code to run as root in the host.
Docker-in-docker is a workaround to make docker work in CI.
Basically a security nightmare and bad design that podman doesn't have.
Kubernetes is a consistent management api for linux (so then you don't need to interact with iptables, mount and all that "hard stuff").
I don't think kubernetes is a solution for context of building an image (a rootfs tree into a .tar.gz file).
Unless you are using kaniko which extends the kubernetes api to add the capability of creating images, but that is handled by kaniko itself via the same api.
I was probably unclear. Kubernetes is a good solution for managing containers (obviously). I use it for CI and it works very, very well, though the CI tools still have more features they could add with the integration.
Any build script can do serious damage to the environment it runs in. Before docker, you'd have to create a new VM from time to time because the build agent had rotted away or died in an altercation with a bad build.
Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it.
I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.
The other option that works really well in a single user environment is to bind to the runner's Docker daemon. That way builds run as siblings of the runner's daemon rather than as children via docker-in-docker.
The huge issue with that is security which is why it's only really practical for a single user or a small group of trusted users. A secondary issue is that (I think) builds can't run simultaneously because they can trample each other when tagging images (since all images are on the runner's daemon).
If I had to build a Docker focused CI system I'd think about using Weave Ignite (AWS Firecracker) to spin up VMs for runners with the Docker socket bound like described above. That way you get all the convenience of binding the Docker socket, but the isolation of a VM that gets thrown away after the build step (or pipeline) finishes. That idea also fits well with local running / debugging IMO because you can bind to the Docker socket on your development workstation (assuming you're not running a large build of parallel tasks which might be an unrealistic assumption).
For us it’s a matter of the CI tool fetching the source code for the docker image, then running docker build, and not necessarily immediately. So you have ‘docker build’ happening toward the end of a set of other tasks. Which I’d really like to have running on a fresh VM or container.
You could separate those into two builds, but the reason they are together is so people think about deployment, and in case any structural changes to the code need to coincide with deployment changes. For instance, breaking changes in APIs. I need a new version of tool/library and I need to change how I call it.
You do know that spinning up a new VM only takes a few seconds? With projects like https://firecracker-microvm.github.io/, the difference between launching a new Docker container or a new VM is negligible.
This works great if you own or rent the hardware, but most cloud providers don't allow nested virtualization.
The cost is not spinning up the vm, it’s maintaining the images. Docker composability reduces the combinatorics problem to a dull roar, and democratizes some of the maintenance effort. You want an image with the bug fix from the latest point release of python? And you need it by noon? Knock yourself out.
Although there are tools to convert docker images to vm images. I expect if I were running community CI infrastructure, getting really familiar with those would be high on my priority list.
Okay, I want to build a container image using gitlab CI, which runs builds in docker. How would you like me to build an image without using docker in docker, or buildah in docker?
We use kaniko[1] in Gitlab CI and it’s working great for us. It’s annoying the kaniko image requires us to specify the entrypoint. There’s some peculiarities with Dive [2], but otherwise it’s been a very easy migration.
Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO.
Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code?
IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too.
I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable.
Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.
People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF?
> I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.
I want this framed or sewn onto a pillow or something.
It's amazing what we can build, it's baffling what we have built.
> Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler
No, the idea is to build and push images you can test directly afterwards in the same conditions. With cache and such, build times shouldn't be too long
Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally.
If you have long CI times, that can hinder development productivity and should be improved as much as possible, or a local replica of the CI needs to be created.
It is really working for you?
Presumably you setup kaniko to build the images first that then run the second part of your pipeline. To run the images, you need to tag them with something that you use inside the gitlab-ci.yml, right?
Now what happens when two people push code that make changes to the containers at the same time?
Your problem is the assumption that a gitlab CI process requires a docker image. If you do the following, you are fine:
* Switch to a shell runner
* Put the CI dockerfile into your repo
* Provide an entry script for CI that builds the container on-demand (and manages caching/cleanup) and then runs the tests/whatever inside that container
The point here is that docker/podman provide you with everything you need as long as you have full control. By using gitlab's default CI, you relinquish this control.
But how would you run integration tests between multiple docker containers? We launch our services in these docker containers and do some integration / e2e tests, and this very much requires Gitlab CI to launch docker containers while inside a Docker container.
After using dind for some time we chose to just mount /var/run/docker.sock and keep using the host machine’s docker instance (mostly for the cache), but all in all dind was working fairly well.
To be honest, to say “you shouldn’t be doing that” is missing the point; one should be able to do anything they want. In my opinion, the root cause here is docker’s client/server model, which is fixed by other container runtimes such as podman and rkt (which unfortunately is deprecated). One should be able to just launch containers as if they were just another process.
To the contrary. You should never expect to be able to do what you want with a system provided by someone else. Gitlab has its design decisions in their default setup and you need to take control to cater your use case. This herd mentality of "everyone is doing it this way" is really fundamentally problematic.
Nowhere in my post am I saying "everyone is doing it this way".
You started out with asserting "you're too far into Docker", we bring up valid use cases for docker-in-docker, and then you saying "This herd mentality [..] is really fundamentally problematic" is really not adding a lot to the discussion.
No one brought up valid use cases for docker-in-docker. They brought up the issue that gitlab mandates docker as an interface (which I totally understand, btw.).
For instance the "how do I compose multiple docker containers" is trivial when you can just execute a script that runs docker or podman. If you really want, you can use docker-compose.
That is exactly my point. Gitlab uses docker runners because it is much simpler for them. But why should you be constrained by what's simpler for gitlab?
Our org just adopted Makisu[1] which works beautifully in our GitLab CI pipelines. After a week of setup and fiddling with the settings, we were able to migrate all our build jobs to Makisu and haven’t looked back. Build times are great too, especially if you set it up with a Redis cache.
Docker and Nix are mostly orthogonal technologies though. Nix is an excellent build tool while Docker is useful as an universal software distribution format. They're really useful together because Nix is actually good at creating compact Docker images. But of course, if you don't have to worry much about distribution, it would be much nicer to stick with Nix as you've mentioned.
Well, in the case of Mac, basically performance, as you still need linux libs, kernel API and everything. But it's much lighter-weight than something like Virtualbox, VMWare, etc.
In Linux iirc, a hypervisor can share such resources with the host system (since they are both Linuxes).
This might be biased on my part but experiences like this is why I tend to openly avoid most Red Hat projects in general. There are definitely some great projects like RHEL, Ansible that are nice to work with.
Even professionally, I tend to steer clear of immature Red Hat projects.
We evaluated Keycloak but went with a vendor solution. OpenShift I believe was also evaluated at my firm and hit a dead-end.
Openshift is a great solution for a k8s setup on prem. From my experience it doesn't bring a whole lot when you're already on the cloud. All the additional services openshift provides, are already available as part of the major cloud offerings.
Openshift years ago at least (last time I looked) - SHUDDER! I couldn't believe it was what big business / enterprise was standardizing on as best in class. It looks like it's now Kube / Docker connected, but back then I think it did it's own thing?
The cost in time / $ / config to maintain and operate at the time was not pretty (all this no doubt has changed a lot).
They gave you three pods for free until they moved to Openshift 3. I ran an entire startup five years using them without paying a single penny. A nice Heroku alternative. (Iot control server, so basically no load)
Openshift was very generous to startups, I can back that up as well.
It always seemed like added complexity to me in exchange for free hosting. I liked the idea in concept more than in practice. But we also weren't big enough to really justify it so eventually we switched to simpler VPS hosting.
Spend too much with Keycloak lately - it's more like a framework with some basic batteries included and you'll hit into corner cases pretty fast. If you are fluent with Java, their templating engine and all the other funky stuff it's a powerful tool - but i.e. their REST-API is bad. List all users including groups? That'll be an additional HTTP call for every user.
Their tools solve complex problems and are free to use - so I'm thankful that they exist - but I can't help but think that there is some lack of elegance&design that causes lot's of complexity - there is no "unix philosophy" to abuse different tools or components to solve problems - it's more like either you'll use the high level APIs with certain non-obvious assumptions (i.e. worked for us, good luck :) or feel free to hack on it if you grok our complex low-level frameworks and libraries... went after a NetworkManager bug once and it was a tour de force between c, glib, dbus with zero documentation. systemd and Keycloak feel very similiar. Powerful if you fit their usecase - horrible if need to tinker with it. But to be honest I've got no idea how to solve these complex problems otherwise. It's probably the best we can do at the moment. Or are there any non-cloud/non-sass solutions that actually have all the features?
I had the same experience, and I'm not really sure why I can't use docker-compose with podman. If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal?
I did try that and it doesn't work, and podman compose isn't as useable. So I switched back to docker.
>If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal?
Podman is (mostly) a drop-in replacement for docker. However, docker-compose is a separate package from docker that requires explicit installation. So too is podman-compose separate from podman, though unfortunately it still needs more work.
I can explain this. Because docker-compose talks to the docker socket and uses the docker protocol. This was the last piece that we nailed with podman. As of Podman 2.0 with the new REST based API with a Docker compatibility layer, we are very close to just being able to use the docker-compose binary to talk to a socket (ran by systemd) which fires up podman.
It doesn't quite work yet today because we are still implementing some of the REST verbs, but it's close. It's definitely a strategic direction for the roadmap. Stay tuned.
Apologies for the patronising comment, but do you really mean that? Docker in Docker works but is intended for the developers of Docker to debug Docker itself. Usually for running Docker from within a container, you just hook up the Docker client to the TCP port of the Docker daemon running outside the container, which isn't strictly Docker-in-Docker.
I ask this in case you're trying a wildly use case (if you really are running true Docker-in-Docker), or are making an unfair comparison (if you're just using the usual Docker client in container to daemon on host). In the latter case, I must admit I don't know what the idiomatic alternative would be for podman, given that I know nothing about it except that it's daemonless (and even that I only learned by reading your comment).
There are some reasons to use docker in docker beyond debug docker itself. For example, in Jenkins case, select specific proxy settings for the daemon for a specific job without affecting other jobs in the same node.
I also needed to run docker commands from within a container (Jenkins container executing docker cmds on the host it's running on). I don't know how common this is but I had the same use case.
As mentioned elsewhere in the thread, this does not require DinD. You can connect to the host’s Docker daemon. (Other interfaces such as K8S are also much safer.)
Yes it is. The Docker socket - /var/run/docker.sock is owned by root and so anyone that has access to it effectively has root on that host. User namespace mitigate this to some degree.
Yes and I am one of those people who mentioned elsewhere that this does not require DinD. Please reread the question I am actually responding to which is a general questions "What kinds of tasks ...?" The parent is asking what 'problem is the DinD solution is solving for. You also can not connect to the host's Docker socket if you do not manage that host where Docker is running in the first place.
If you don't manage your own CI server then that is not an option is it?
I also never said that it requires two levels of Docker or that it requires DinD. I was responding to the general question of the parent who was asking a question of someone who was running DinD. I responded to the GP below that Kaniko also solves this problem so clearly I'm not advocating for running DinD or that this is even needed.
If you mount docker.sock you're building on the runner's daemon so you have to worry about builds interfering with each other. If you use docker-in-docker you get a clean environment every time.
Also if your application is shipped as orchestrated containers (like docker-compose), or as multiple containers in a 'pod' (e.g. sidecars), you may want the ability to run containers from containers as part of CI.
As of Molecule 3.0, at least -- they didn't in the past.
A while ago, I, unfortunately, decided to add "proper tests" to all of my Ansible roles and decided to use Molecule (2.22, at the time, IIRC). As I don't use Docker, I was using "lightweight VMs" I had created (w/ Packer, converted into Vagrant boxes) with VirtualBox for all of this testing.
I spent I don't know how many hours across several days learning the "toolchain", getting everything setup and working properly, adding full test coverage, and so on. Not long afterwards, they released Molecule 3.0 which required using Docker. :/
“kind” is a example in my day to day, how do I run an ephemeral kube cluster in ci to deploy and test my app against without dind/kind? I want N clusters running on 1 host to support many CI jobs.
We use docker-in-docker as well to deploy our Jenkins workers. It has it's pros (running Ubuntu 18.04 and Ubuntu 20.04 workers side-by-side and ease of pushing out an update) but it's fragile and confusing. Things can get really hacky when you need to start sharing volumes or injecting additional configuration.
I'm much more confident with packer now though. Next time I do any major work on our Jenkins infrastructure I'm ripping out docker-in-docker for the workers and replacing it with packer built images.
> but is intended for the developers of Docker to debug Docker itself
Heh. We use DIND with docker compose to have a container which has KUBERNETES inside. And it even works.
How's that for a wild use-case?
EDIT: That's done to create a local dev environment, with K8s, localstack, infra, etc. Instead of having multiple machines or deploying everything outside containers.
I do a lot of build automation work, and running containers within containers would be super useful for me.
That said docker-in-docker doesn't work without running privileged or forwarding the host port.
It's a non-starter for me, there are the obvious security problems, but also practical non-security issues.
Forwarding the port causes encapsulation issues, a build job can finish leaving stuff running, can also interfere with other jobs on the system.
Using privileged containers isn't an option when using things like ECS fargate.
I think what most people mean when they say docker-in-docker is in layman terms. You simply just a way to do "docker run" from within another container. Whether that's done by actually running docking daemon inside or mounting something from the host or whatever else i don't really care. I just want it to work.
Sure, there are some details you might want to control, like image caches and such being shared with host, i just find there is a lack of documentation and best practices of how to do nested docker, if that is even intended to work or if mounting docker.sock is an unsupported hack. Most information found about this is scattered on shady blogs.
For the examples of CI brought up the use cases are fairly obvious, you have a jenkins installation with x plugins installed - running as a container. Within this jenkins you are building multiple different projects which all require their own respective image to build. As a project developer here i don't even want to know if this jenkins is a bare metal, a vm or a container. Here docker is more used to bundle all the dependencies, not for strict security with perfect containerization.
Your life will be a lot less painful if you abandon docker-in-docker. The Jenkins solution is to simply execute containers using the host's Docker daemon, or via a container orchestration system which you call out to.
The latter is the best option as it's much more scalable and doesn't require ugly proprietary hacks. Docker in Docker requires vertical scaling and more complex management for intermediate states/maintenance.
Podman probably isn't ready to replace Docker yet, but rootless containers are the easy-mode for federation of clusters. If you can supply the other features you need (and I think most can) it's probably worth it in the long run.
We're working hard on docker-compose support. With the new Podman 2.0 REST API with a Docker compatibility layer, we are close. Not quite there, but definitely strategically on the roadmap.
If you are using Docker-in-Docker, you may want to checkout the new Sysbox runtime (find it on Github). It's a new type of runc that sits below Docker and creates rootless containers capable of running Docker, systemd, K8s, etc. All you have to do is "docker run --runtime=sysbox-runc" <some-image-with-docker> and you'll get a docker daemon that is fully isolated from the host. It's a great way of avoiding privileged containers or mounts to the host docker socket.
Is the Docker daemon actually a huge security hole?
I see people championing Podman because it's daemonless but is it actually beneficial or is it championed because it's Red Hat and a case of security check boxing?
Do you mean in that adding a user to the docker group is effectively handing them root? Because if you don't do that, then running docker requires root/sudo, which means that it should be exactly as secure as anything else
It's a root daemon which can be made to run arbitrary commands by anyone using its interface, so yes it is a huge, huge security burden. You can't just brush that off as a superficial problem.
Given that the most common case is running docker in server environments in VMs and the sysadmins are root, is this a real issue? Can you tell me an environment where a multi-tenant system runs docker?
Yes, it[1] is[2] a[3] real[4] issue[5] which leads to privilege escalation bugs. It also doesn't help that most containers that Docker is responsible for managing is supposed to be unprivileged, and Docker itself is commonly used as a component for a multi-tenant container runtime.
I love the idea of Podman being daemonless and (potentially) rootless. Regardless of the presence of security flaws in dockerd, the Podman architecture just feels cleaner.
And the CLI compatibility is great. Until it isn't. At work we switched to Podman for a small deployment because Docker didn't yet work with cgroupsv2 and many hours were spent debugging Podman-specific issues. In the end switching to cgroupsv1 would have been significantly less work.
Therefore claiming you can `alias docker=podman` is a bit disingenuous. You can, but only if you don't do anything Podman can't handle and what it can and cannot handle isn't immediately obvious.
All that said, I wish the project the best and hope it reaches the maturity where this alias actually does work.
I'm afraid I didn't and I cannot really recall, but many were related to networking. I think one was that I couldn't have two containers on the same network listen to the same port, another one was related to Podman not playing nice with nftables, but the specifics elude me.
EDIT: Upon further reflection, I think Docker doesn't really work with nftables either, so that one isn't on Podman. It just so happened we made that switch at the same time. Regardless, there were other problems. I'll check to see if I can find any records of the problems later.
Last time I tried podman it looked very cool being rootless and daemonless, except that since it's rootless I couldn't create the necessary network interfaces... making it pretty much worthless.
If GP thinks the selling point of podman is that it doesn't have to run as root, but to actually do anything meaningful they have to run it as root after all, why would they use podman?
Many people say that, but don't feel like it's very important.
Having a deamon or not is a technical detail that most people do not care about in my opinion. And it has advantages too, like accessing Docker remotely or from another VM on the same host, or directly from the host which is nice for Docker on Mac or Windows.
I find the podman integrates with systemd well claims a bit dubious.
Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background and do the process supervision.
I found this very surprising. I still don't understand why they made that choice.
I find the podman integrates with systemd well claims a bit dubious.
Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background.
I found this very surprising. I still don't understand why they made that choice.
At least they do play nice with the whole "systemd owns the Cgroups tree" story. A thing that was always a bit painful with docker.
Fun note. Systemd-nspawn actually can run OCI containers directly as well these days. However I'm not sure if it's feature-complete
We've tried this (podman 2.0.5) and have hit some really really annoying bugs:
- Build layer cache doesn't seem to work. If I rebuild locally with podman, it correctly detects cache hits and the build is fast. On our Jenkins server (RHEL 8) with podman 2.0.5 it doesn't. It randomly doesn't cache hit, causing builds to take 20x longer than with docker CE.
- Podman is insanely slow at building images in general. COPY emptyfolder/ /emptyfolder/ takes 2 seconds. We have dozens of things to COPY and it's stupid slow compared to docker CE. Buildah doesn't seem any better.
- Systemd integration has bugs. If you use the default generated systemd unit file, it does not kill processes when exiting and leaves them dangling. Even after removing the strange KillMode=none it says to put in there, it still leaves processes dangling. Podman sometimes loses track of the container. It will list nothing in "podman ps" but the processes will still be running.
Not using --cache-from, though I did try the various caching command line options to try to troubleshoot. Just a vanilla Dockerfile with a few RUN commands and COPY commands. It would hit cache until about the third layer, and then would cache break every time.
On my local computer (Arch) podman is v2.1.1, which seems to have whatever bug I was hitting fixed.
So I guess my complaint isn't about podman specifically-- It had bugs and they were fixed, and that's great. But I hate that RHEL 8 touts it as a docker replacement, and won't carry docker in their repositories, when the version they have in their production releases is so broken.
We eventually sledgehammered docker CE's CentOS repo into our RHEL 8 jenkins server and now everything works perfectly.
Running podman on the production webservers seems to work okay though-- apart from the process killing problems.
I've said this so many times, but we wanted to carry Docker in RHEL 8, but we couldn't. We would have been forced to use Moby and it's not a well contributed to project anymore. The whole Docker contributor community has fallen apart, and Red Hat needs to be able to support a container engine until the mid 2030s.
You can get access to Moby in Fedora, but it just wasn't viable to include docker in RHEL 8 for both legal and community risks.
Podman, buildah, and skopeo are insanely awesome tools that let you transition an image from a public registry to your own private deployment in a jiffy. The article doesn't mention it but podman can also dump a pod configuration you can use in kubernetes and open shift/okd so if you're not a yaml wizard you can still get stuff running containerized.
Lack of a drop-in docker-compose replacement is what stopped me last time I tried podman. podman-compose looked decent but it lacked a bunch of basic features and didn't work smoothly last time I tried it.
Assuming that it's Red Hat's fault that there is no el8 package for Docker, I think Red Hat is really shooting themselves in the foot by not supporting Docker in RHEL8. (You can still install the el7 version of Docker on el8 by performing an enable/disable/enable/disable/... module dance, but... why isn't there just an el8 version?)
Yes, but not Docker. (Also Podman is developed by Red Hat.) I think they have rather high hopes there, and are more likely to just shoot themselves in the foot by being one of a small number of distros that don't have Docker. In fact, rebuilders of RHEL (e.g. Oracle Linux) should maybe consider explicitly supporting Docker. "We are 100% compatible with RHEL but additionally also support Docker" definitely makes for better marketing copy than just "We are 100% compatible with RHEL".
Nope, it's still not there, simplest things work but as soon as you diverge to something more complex it fails.
I tried using it as Docker replacement, but various tools that use docker (using dockerized pip in serverless framework) and complex docker-compose files (dockerized Magento) were broken.
I always like to chime in and recommend people check out Singularity[0] containers if they haven't before. It's more common in academia because you can run containers without privileges which is nice in HPC environments. The containers themselves are simple immutable flat files that can be easily copied around, backed up, etc. There's also no system daemon. It's nice knowing everything you need for your container system exists in the containers themselves and a single singularity executable. No volumes, remembering to append --rm to avoid dangling containers, wondering where the actual images are stored, etc. Of course there are tradeoffs but I like them for certain use cases.
After reading this thread, I realize that the 4 GB (average) of my vagrant box images wasn't that bad in retrospect, and that loaded a fully baked OS with packages from apt and just required VirtualBox and its extension pack. (this was not an optimized setup, I was such a noob at the time, just trying to wrangle our developer tools)
The only reason I moved to docker years ago was I wanted a tighter reproducible workflow (ie docker-compose up) for all other developers, turns out vagrant has (at some point in the last 5 years) solved this problem too, with packer (I didn't know about this 5 years ago for whatever reason)
Makes me long for going back to that. At least once you built your base image, it was done and you could just make fast linked images from there. I bet with Alpine you could get a vagrant up in a few minutes tops, then it boots in seconds.
Never liked Ruby as a config language, but doing complex things like setting up shared networks and folders was a breeze comparatively I felt.
Never used vagrant in a production capacity though. Everywhere I ever worked always deployed to bare metal or essentially we bought our services (cloud functions, semi-managed containers etc)
I think there's a disagreement about what Docker-in-Docker actually is. Some people think it's running a docker daemon inside a container, and others think it's accessing the host docker daemon from inside a container by mounting the docker socket as a volume
It's compatible with cgroups v2 unlike the standard Docker. If you're using Fedora, you have to add a kernel parameter to Grub to use cgroups v1 instead.
RedHat seems to be pushing a standard ecosystem for Linux: systemd, Wayland, SELinux, GNOME, and now maybe podman. I've been on Linux for a while; it's a welcome change from all the fragmentation I'm used to. Whereas others try to work around the kernel and implement their own things in parallel (see: Canonical's AppArmor, LXC, OpenZFS), RedHat just goes with what Linux already has like SELinux/cgroups v2/btrfs, which I think is more likely to last and just feels better. If RedHat goes away, I'm fine since I'm ultimately only relying on Linux features. If Canonical goes away, then I'd have to switch to a different stack. That's probably why government, enterprises, Amazon, etc. still prefer RedHat.
Anecdotally, I've worked with developers at Redhat and Canonical, and the Redhat developers had passion. They really believed in open source and the linux community. In comparison Canonical seemed like Just Another Software Company to me.
If you want to use btrfs on RHEL8, you still can: go to https://elrepo.org/ and install elrepo-release, then install kernel-ml from their -kernel repo and btrfs-progs from their -testing repo.
Running an unsupported kernel doesn't mean that support won't help you at all. It just means they won't help you with kernel-related issues. There's still a lot of value in support for the rest of the system.
Both of you are right. Red Hat pays for Fedora in a lot of ways, but does not control it. There are obviously a lot of personal and professional relationships that gives Red Hat influence over Fedora, but the purse strings are never used to control it.
I think governments prefer redhat because they are consistent and they accept gobs of cash to promise security patches to software that are past their support window.
I don't know about the gobbles of cash, but I'm pretty sure Canonical supports Ubuntu releases way past EOL for paying customers - they call it the ESM (Extended Security Maintenance), which 'provides important security fixes for the kernel and the most essential user space packages in Ubuntu'.
It tends to also be in the opposite direction, big expensive enterprise software is supported on redhat forever and ever. And the issue isn’t just security updates, but full software and hardware support for the same stack for many years.
Lots of people don’t want to recreate their stack every few years for the latest cool thing.
Depends on your definition of support I suppose. Redhat employs so many more low level engineers they can generally get you a z stream patch or something to fix a legitimate bug. Canonical is an exceptional marketing company, but I've found their engineering to be lacking when it comes to enterprise. As someone who's worked as a linux monkey for companies who pay Canonical for support and for companies who pay Redhat for support, I've always found the Redhat support to actually solve the root cause. When we filed what we thought were legitimate bugs with Canonical, the common response was just to "upgrade to the latest version of Ubuntu". When we asked them to confirm that doing so actually fixed our problems we just got crickets.
Governments also prefer Redhat because they early on and continuing today have shown a commitment to the market by investing in getting their products certified to meet government required standards including developing the required capabilities when not already available in the Linux ecosystem, developing documentation and software to assist deploying systems in a manner consistent with government security requirements and participating in conferences, working groups, interoperability tests and other government targeted or sponsored forums.
Some other Linux vendors now also do so but started much later and even now do so to a much lesser degrees.
More like, Redhat/IBM defines what Linux has simply by paying most Linux devs, and of course they take advantage of it in userspace and sys management.
cgroups/namespaces, systemd, gnome (and it's integration with systemd) plus others are two-sided swords: one one hand they create new functionality, but on the other hand the price to pay is concentrating Linux know-how in one place, and greatly diminishing portability of Linux apps vs other Unix O/Ses and even other Linuxes that don't want to go with the program of absorbing ever more functionality into kernels and system frameworks for no real reason other than monopolization.
Unix was designed as a simple portable operating system in very short time. With 30 years of development, the situation today could also be interpreted in such a way that Linux devs just can't stop to add code. Time will tell if Linux can be maintained if the original generation of devs step down. I'd feel more comfortable if we've let kernels stay minimal rather than becoming kitchen sinks. Apart from better portability, this would've also enabled younger devs to start from scratch rather than having to maintain daddy-o's monstrosity.
As a 23 y/old younger (well, young-ish) dev, it strikes me that this isn't anything new. Old commercial solutions since nearly the beginning have been predicated on proprietary solutions specific to some vendor's UNIX based OS. HP-UX, AIX, even back to the really early commercial releases of v7 all seem to have a common theme of having that unix core but with a lot of nice features added on, typically with pretty insane and convoluted frameworks. Tru64's clustering, AIX's management interface, all the competing UIs and management interfaces. All nice features deeply integrated into a proprietary solution as a form of vendor lock-in.
To your point, it seems like eventually this ends poorly for the incumbents, when they become too bloated and too difficult to extend or modify, since you have to work around some over-engineered high-availability process clustering to touch anything.
In Linux's case, I think the kernel itself is actually fine, but the user land is increasingly tightly glued to an increasingly convoluted set of RedHat stacks.
I mean, it always has -- RedHat made the modern posix threads library, Linux's PAM, and several other core parts, but at least those were pretty modular and "unixy", by comparison to the monolith being developed now.
>and greatly diminishing portability of Linux apps vs other Unix O/Ses
Thank's god for that! An OS should build on its strenghts, not be a least common demoninator for portabillity's sake. If that was ok we might just as well just use one OS
Indeed, for me the best UNIX experiences that have been created so far have been NeWS and NEXTSTEP (macOS), and I wouldn't mind if Linux would just standardize on one stack as well.
What you are noticing is the RedHat “upstream first” policy, and it’s very practical. They try to minimize RedHat specific modifications and instead contribute improvements directly to the original projects.
This reduces the amount of code they are responsible for maintaining, and everyone involved gets the benefit of open source collaboration.
277 comments
[ 3.0 ms ] story [ 293 ms ] threadWow that's a great adoption hack; just alias another program as another
The conclusion that I have to reach is that there are more docker users on Macbooks than I realized.
And of course there's a Podman Machine: https://developers.redhat.com/blog/2020/02/12/podman-for-mac...
And systemd actually comes with two image launching systems systemd-nspawn and systemd-portable. And then with systemd-machined you can add software that needs virtualization too.
The interface to journald is more complicated than it should be but it’s also really powerful — docker logs doesn’t hold a candle to the kind of filtering it can do.
Overall systemd is a stupidly powerful and featureful supervisor compared to Docker. Just the dependency management alone should demonstrate that. Then you get mounts, swap, socket activation, more powerful restarting policy and the whole suite of isolation and security features.
I thought the point of logging to stdout (i.e. docker logs) is that you just take that output and dump it to another server for processing and filtering.
journald/journalctl seem to be a solution a few decades late to the party. For a single user machine or a single app prod environment, I would take a plaintext log any day of the week. At least I can remember how to grep the damn thing. And then when you get to a distributed system, what's the point of journalctl? You would hopefully have all of that logging aggregated together in one place with a much nicer web interface for it all.
The big ease-of-use win for journald is that it captures process stdout. No need to run daemons in the foreground ever.
Note also that if I just want the log stream, I have to pipe through less to get the full log messages. There’s also a flag for it, but I can’t remember my workaround is easier than digging through man pages.
Again, no big deal, just friction. Like everything in the systemd ecosystem—everything is manageable, but it’s tedious.
Maybe it already has something like this, but I feel like systemd could provide an API endpoint for applications to send a simple status when they are "ready" - at which point it would be up to the developers to provide that at the right time.
https://docs.docker.com/engine/reference/builder/#healthchec...
Perfect. Thank you!
Docker was created to give an extra life for legacy applications depending on outdated packages, the concept of gluing all dependencies withing a compressed rootfs and shipping that, with low effort.
Docker became popular because of that, but seems that did more harm than good for the ecosystem in terms of security, that why podman might have a great future. The docker interface is bad and comparing that with systemd is quite a stretch. ;)
Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.
Aren't containers just processes with namespaces and cgroups?
I suppose a more secure runtime doesn't hurt.
I don't know, systemctl is always a pain to interact with. The Docker CLI is pretty intuitive--you have different resources and different verbs for interacting with those resources. This includes logs. No need to use a separate (and confusing) tool nor dig through man pages. I'm of the philosophy that it's better for a tool to be intuitive than obscure-but-has-manpages.
> Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.
I guess I meant that INI isn't a standard file format--different parsers behave differently, and structured data is often coded in strings in some bespoke format.
> Aren't containers just processes with namespaces and cgroups?
My mind isn't made up that containers are the ideal process unit, but I do like some things about them (and the container ecosystem more generally). That they come with their dependencies bundled is pretty nice, but I think the toolchain needs to improve to mitigate security concerns and so on. Something like Nix would be helpful, but Nix also has its own problems. Again, it's directionally correct. In any case, users don't need to be managing their own namespaces and cgroups.
at some point systemd is an embrance-and-extend formula to control the linux ecosystem
There is, frankly, a lot of appeal to me in a simple INI file that starts up something I unzipped.
Work also uses a special docker RUN command that can use the host's ssh keyring to, eg. Install something from a private github repo. Doesn't seem to exist in podman.
That is amazing; thanks for pointing it out:)
At work we dont get root access so I thought it would be perfect but sysadmins couldn't figure out how to configure either so was bust there too.
One thing I want to point out though for anybody not familiar with the differences between podman and docker, for the most part alias docker=podman will "just work" except for these situations:
1. docker-compose. podman-compose attempts to cover this but I've heard it's not quite there yet
2. Mounting the docker socket into the container. Podman is daemonless which means that won't work. There is work going on right now to allow Podman to be driven in a similar way if needed, but I recently tried to set it up and hit a bug[1]. CRI-O brings a daemon, and it is used extensively in Kubernetes and OpenShift, but not so much outside of that.
[1] https://github.com/containers/podman/issues/8323
On the other hand, I don't think you fully understand where I'm coming from. "Optimizing for stagnation", "stuck on the same plateau". Idk, I just want to run a container. I'm totally fine being on the same Docker plateau if something like Podman makes me re-learn what I already know how to do.
Unlike Git, which, from the start, was different and BETTER than SVN, can you really make the case that Podman would be BETTER if it didn't try to follow Docker conventions? If not, what are we talking about here?
I've swapped back to docker on Fedora and CentOS by forcibly installing docker on those unsupported platforms because podman-compose doesn't work, containers just didn't work like expected in several situations (forgot detail) and portainer wouldn't work against it.
Either they should have sped up the podman development or shouldn't have deprecated docker at this point. Maybe RedHat just didn't like a third party taking the stewardship in the container business.
The lack of filesystem isolation and volume support are the last things keeping me from jumping ship.
When we shifted to RHEL 8, we attempted to move this over to Podman and it went miserably (this was back in November 2019). The main reason being is that podman-in-podman doesn't work and had bugs (at least back in Nov 2019). Maybe it fixed now but this was our experience. We ended up doing quite a bit of analysis on podman only to conclude it's simply not there yet relative to docker (ecosystem and ergonomics).
There are quite a few corner cases that docker quite simply supports out of the box beautifully that podman doesn't support or just has bugs.
I like the what the project is trying to solve by being daemonless, but this is not as simple as a drop in replacement for docker that RedHat markets it as (alias docker=podman).
We ended up sticking to docker professionally and personally, I am still using docker over podman. The ecosystem and ergonomics are just far too nice to give up over podman.
I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.
In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.
my beastly 12GB image that even includes Matlab wants a word with you
Perhaps in the next 10 years we will be rediscovering packages. :P
If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
True, that.
To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
tl;dr pip silently breaks my environments, mostly connected to upgrading numpy and other scientific/data science libraries.
Docker-in-docker is a workaround to make docker work in CI.
Basically a security nightmare and bad design that podman doesn't have.
I don't think kubernetes is a solution for context of building an image (a rootfs tree into a .tar.gz file).
Unless you are using kaniko which extends the kubernetes api to add the capability of creating images, but that is handled by kaniko itself via the same api.
Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it.
I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.
The huge issue with that is security which is why it's only really practical for a single user or a small group of trusted users. A secondary issue is that (I think) builds can't run simultaneously because they can trample each other when tagging images (since all images are on the runner's daemon).
If I had to build a Docker focused CI system I'd think about using Weave Ignite (AWS Firecracker) to spin up VMs for runners with the Docker socket bound like described above. That way you get all the convenience of binding the Docker socket, but the isolation of a VM that gets thrown away after the build step (or pipeline) finishes. That idea also fits well with local running / debugging IMO because you can bind to the Docker socket on your development workstation (assuming you're not running a large build of parallel tasks which might be an unrealistic assumption).
You could separate those into two builds, but the reason they are together is so people think about deployment, and in case any structural changes to the code need to coincide with deployment changes. For instance, breaking changes in APIs. I need a new version of tool/library and I need to change how I call it.
This works great if you own or rent the hardware, but most cloud providers don't allow nested virtualization.
Although there are tools to convert docker images to vm images. I expect if I were running community CI infrastructure, getting really familiar with those would be high on my priority list.
Userland docker builds
[1] https://github.com/GoogleContainerTools/kaniko
[2] https://github.com/wagoodman/dive/issues/318
Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO.
Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code?
IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too.
I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable.
Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.
People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF?
I guess I got on a bit of a rant...
I want this framed or sewn onto a pillow or something.
It's amazing what we can build, it's baffling what we have built.
No, the idea is to build and push images you can test directly afterwards in the same conditions. With cache and such, build times shouldn't be too long
Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally.
If you have long CI times, that can hinder development productivity and should be improved as much as possible, or a local replica of the CI needs to be created.
In fact, they already provide a utility for doing this: gitlab-runner exec[0].
[0]: https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
Now what happens when two people push code that make changes to the containers at the same time?
* Switch to a shell runner
* Put the CI dockerfile into your repo
* Provide an entry script for CI that builds the container on-demand (and manages caching/cleanup) and then runs the tests/whatever inside that container
The point here is that docker/podman provide you with everything you need as long as you have full control. By using gitlab's default CI, you relinquish this control.
After using dind for some time we chose to just mount /var/run/docker.sock and keep using the host machine’s docker instance (mostly for the cache), but all in all dind was working fairly well.
To be honest, to say “you shouldn’t be doing that” is missing the point; one should be able to do anything they want. In my opinion, the root cause here is docker’s client/server model, which is fixed by other container runtimes such as podman and rkt (which unfortunately is deprecated). One should be able to just launch containers as if they were just another process.
You started out with asserting "you're too far into Docker", we bring up valid use cases for docker-in-docker, and then you saying "This herd mentality [..] is really fundamentally problematic" is really not adding a lot to the discussion.
For instance the "how do I compose multiple docker containers" is trivial when you can just execute a script that runs docker or podman. If you really want, you can use docker-compose.
We're using podman in containers inside gitlab ci.
We're _also_ running tests of containers inside containers in containers using gitlab-ci.
The main workaround we've applied is using crun as the runtime rather than runc.
[1]: https://github.com/uber/makisu
I'm not sure why it doesn't do this by default. Performance I guess.
I use docker for most of my clients' work but for in house stuff I just use nix.
That has not been the case for a good while now... Docker has been running directly on a hypervisor on the Mac.
In Linux iirc, a hypervisor can share such resources with the host system (since they are both Linuxes).
Even professionally, I tend to steer clear of immature Red Hat projects.
We evaluated Keycloak but went with a vendor solution. OpenShift I believe was also evaluated at my firm and hit a dead-end.
The cost in time / $ / config to maintain and operate at the time was not pretty (all this no doubt has changed a lot).
It always seemed like added complexity to me in exchange for free hosting. I liked the idea in concept more than in practice. But we also weren't big enough to really justify it so eventually we switched to simpler VPS hosting.
Their tools solve complex problems and are free to use - so I'm thankful that they exist - but I can't help but think that there is some lack of elegance&design that causes lot's of complexity - there is no "unix philosophy" to abuse different tools or components to solve problems - it's more like either you'll use the high level APIs with certain non-obvious assumptions (i.e. worked for us, good luck :) or feel free to hack on it if you grok our complex low-level frameworks and libraries... went after a NetworkManager bug once and it was a tour de force between c, glib, dbus with zero documentation. systemd and Keycloak feel very similiar. Powerful if you fit their usecase - horrible if need to tinker with it. But to be honest I've got no idea how to solve these complex problems otherwise. It's probably the best we can do at the moment. Or are there any non-cloud/non-sass solutions that actually have all the features?
I did try that and it doesn't work, and podman compose isn't as useable. So I switched back to docker.
Podman is (mostly) a drop-in replacement for docker. However, docker-compose is a separate package from docker that requires explicit installation. So too is podman-compose separate from podman, though unfortunately it still needs more work.
It doesn't quite work yet today because we are still implementing some of the REST verbs, but it's close. It's definitely a strategic direction for the roadmap. Stay tuned.
Apologies for the patronising comment, but do you really mean that? Docker in Docker works but is intended for the developers of Docker to debug Docker itself. Usually for running Docker from within a container, you just hook up the Docker client to the TCP port of the Docker daemon running outside the container, which isn't strictly Docker-in-Docker.
I ask this in case you're trying a wildly use case (if you really are running true Docker-in-Docker), or are making an unfair comparison (if you're just using the usual Docker client in container to daemon on host). In the latter case, I must admit I don't know what the idiomatic alternative would be for podman, given that I know nothing about it except that it's daemonless (and even that I only learned by reading your comment).
I also never said that it requires two levels of Docker or that it requires DinD. I was responding to the general question of the parent who was asking a question of someone who was running DinD. I responded to the GP below that Kaniko also solves this problem so clearly I'm not advocating for running DinD or that this is even needed.
This way you don't need to grant "docker" group to the "ci" user and you avoid having your cluster compromised one commit away :)
Also if your application is shipped as orchestrated containers (like docker-compose), or as multiple containers in a 'pod' (e.g. sidecars), you may want the ability to run containers from containers as part of CI.
A while ago, I, unfortunately, decided to add "proper tests" to all of my Ansible roles and decided to use Molecule (2.22, at the time, IIRC). As I don't use Docker, I was using "lightweight VMs" I had created (w/ Packer, converted into Vagrant boxes) with VirtualBox for all of this testing.
I spent I don't know how many hours across several days learning the "toolchain", getting everything setup and working properly, adding full test coverage, and so on. Not long afterwards, they released Molecule 3.0 which required using Docker. :/
https://github.com/kubernetes-sigs/kind https://github.com/bsycorp/kind
https://github.com/GoogleContainerTools/kaniko
I'm much more confident with packer now though. Next time I do any major work on our Jenkins infrastructure I'm ripping out docker-in-docker for the workers and replacing it with packer built images.
Heh. We use DIND with docker compose to have a container which has KUBERNETES inside. And it even works.
How's that for a wild use-case?
EDIT: That's done to create a local dev environment, with K8s, localstack, infra, etc. Instead of having multiple machines or deploying everything outside containers.
That said docker-in-docker doesn't work without running privileged or forwarding the host port.
It's a non-starter for me, there are the obvious security problems, but also practical non-security issues. Forwarding the port causes encapsulation issues, a build job can finish leaving stuff running, can also interfere with other jobs on the system.
Using privileged containers isn't an option when using things like ECS fargate.
Sure, there are some details you might want to control, like image caches and such being shared with host, i just find there is a lack of documentation and best practices of how to do nested docker, if that is even intended to work or if mounting docker.sock is an unsupported hack. Most information found about this is scattered on shady blogs.
For the examples of CI brought up the use cases are fairly obvious, you have a jenkins installation with x plugins installed - running as a container. Within this jenkins you are building multiple different projects which all require their own respective image to build. As a project developer here i don't even want to know if this jenkins is a bare metal, a vm or a container. Here docker is more used to bundle all the dependencies, not for strict security with perfect containerization.
The latter is the best option as it's much more scalable and doesn't require ugly proprietary hacks. Docker in Docker requires vertical scaling and more complex management for intermediate states/maintenance.
Podman probably isn't ready to replace Docker yet, but rootless containers are the easy-mode for federation of clusters. If you can supply the other features you need (and I think most can) it's probably worth it in the long run.
podman-compose doesn't support as much as I needed for my current deployments - although offhand I cannot recall what was missing.
I'm looking forward to trying things again in a few months, but those corner-cases can be real pains to deal with.
I see people championing Podman because it's daemonless but is it actually beneficial or is it championed because it's Red Hat and a case of security check boxing?
[1] https://www.cvedetails.com/cve/CVE-2019-15752/
[2] https://www.cvedetails.com/cve/CVE-2019-14271/
[3] https://www.cvedetails.com/cve/CVE-2019-5736/
[4] https://www.cvedetails.com/cve/CVE-2018-15664/
[5] https://www.cvedetails.com/cve/CVE-2018-15514/
Yes.
Also the containers on docker hub are a security dumpster fire.
And the CLI compatibility is great. Until it isn't. At work we switched to Podman for a small deployment because Docker didn't yet work with cgroupsv2 and many hours were spent debugging Podman-specific issues. In the end switching to cgroupsv1 would have been significantly less work.
Therefore claiming you can `alias docker=podman` is a bit disingenuous. You can, but only if you don't do anything Podman can't handle and what it can and cannot handle isn't immediately obvious.
All that said, I wish the project the best and hope it reaches the maturity where this alias actually does work.
EDIT: Upon further reflection, I think Docker doesn't really work with nftables either, so that one isn't on Podman. It just so happened we made that switch at the same time. Regardless, there were other problems. I'll check to see if I can find any records of the problems later.
Edit: apparently now net4slirp is recommended... Maybe I'll give it another try.
Having a deamon or not is a technical detail that most people do not care about in my opinion. And it has advantages too, like accessing Docker remotely or from another VM on the same host, or directly from the host which is nice for Docker on Mac or Windows.
[1]: https://news.ycombinator.com/item?id=25165789
Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background and do the process supervision.
I found this very surprising. I still don't understand why they made that choice.
Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background.
I found this very surprising. I still don't understand why they made that choice.
At least they do play nice with the whole "systemd owns the Cgroups tree" story. A thing that was always a bit painful with docker.
Fun note. Systemd-nspawn actually can run OCI containers directly as well these days. However I'm not sure if it's feature-complete
- Build layer cache doesn't seem to work. If I rebuild locally with podman, it correctly detects cache hits and the build is fast. On our Jenkins server (RHEL 8) with podman 2.0.5 it doesn't. It randomly doesn't cache hit, causing builds to take 20x longer than with docker CE.
- Podman is insanely slow at building images in general. COPY emptyfolder/ /emptyfolder/ takes 2 seconds. We have dozens of things to COPY and it's stupid slow compared to docker CE. Buildah doesn't seem any better.
- Systemd integration has bugs. If you use the default generated systemd unit file, it does not kill processes when exiting and leaves them dangling. Even after removing the strange KillMode=none it says to put in there, it still leaves processes dangling. Podman sometimes loses track of the container. It will list nothing in "podman ps" but the processes will still be running.
> --cache-from
> Images to utilize as potential cache sources. Podman does not currently support caching so this is a NOOP.
On my local computer (Arch) podman is v2.1.1, which seems to have whatever bug I was hitting fixed.
So I guess my complaint isn't about podman specifically-- It had bugs and they were fixed, and that's great. But I hate that RHEL 8 touts it as a docker replacement, and won't carry docker in their repositories, when the version they have in their production releases is so broken.
We eventually sledgehammered docker CE's CentOS repo into our RHEL 8 jenkins server and now everything works perfectly.
Running podman on the production webservers seems to work okay though-- apart from the process killing problems.
You can get access to Moby in Fedora, but it just wasn't viable to include docker in RHEL 8 for both legal and community risks.
I tried using it as Docker replacement, but various tools that use docker (using dockerized pip in serverless framework) and complex docker-compose files (dockerized Magento) were broken.
[0]: https://sylabs.io/
The only reason I moved to docker years ago was I wanted a tighter reproducible workflow (ie docker-compose up) for all other developers, turns out vagrant has (at some point in the last 5 years) solved this problem too, with packer (I didn't know about this 5 years ago for whatever reason)
Makes me long for going back to that. At least once you built your base image, it was done and you could just make fast linked images from there. I bet with Alpine you could get a vagrant up in a few minutes tops, then it boots in seconds.
Never liked Ruby as a config language, but doing complex things like setting up shared networks and folders was a breeze comparatively I felt.
Never used vagrant in a production capacity though. Everywhere I ever worked always deployed to bare metal or essentially we bought our services (cloud functions, semi-managed containers etc)
So many setups would be achievable with simple ifs.
1) why the hell would you want Docker-in-Docker
2) I can’t live without Docker-in-Docker
RedHat seems to be pushing a standard ecosystem for Linux: systemd, Wayland, SELinux, GNOME, and now maybe podman. I've been on Linux for a while; it's a welcome change from all the fragmentation I'm used to. Whereas others try to work around the kernel and implement their own things in parallel (see: Canonical's AppArmor, LXC, OpenZFS), RedHat just goes with what Linux already has like SELinux/cgroups v2/btrfs, which I think is more likely to last and just feels better. If RedHat goes away, I'm fine since I'm ultimately only relying on Linux features. If Canonical goes away, then I'd have to switch to a different stack. That's probably why government, enterprises, Amazon, etc. still prefer RedHat.
On the other hand isn’t btrfs becoming the default in Fedora 33, thus eventually making a comeback in RHEL?
It's also noteworthy that the BTRFS change was driven by community members, mainly from Facebook.
So if you want to use Btrfs, you might as well use it on something besides RHEL.
https://www.redhat.com/en/topics/linux/fedora-vs-red-hat-ent...
Lots of people don’t want to recreate their stack every few years for the latest cool thing.
Some other Linux vendors now also do so but started much later and even now do so to a much lesser degrees.
More like, Redhat/IBM defines what Linux has simply by paying most Linux devs, and of course they take advantage of it in userspace and sys management.
cgroups/namespaces, systemd, gnome (and it's integration with systemd) plus others are two-sided swords: one one hand they create new functionality, but on the other hand the price to pay is concentrating Linux know-how in one place, and greatly diminishing portability of Linux apps vs other Unix O/Ses and even other Linuxes that don't want to go with the program of absorbing ever more functionality into kernels and system frameworks for no real reason other than monopolization.
Unix was designed as a simple portable operating system in very short time. With 30 years of development, the situation today could also be interpreted in such a way that Linux devs just can't stop to add code. Time will tell if Linux can be maintained if the original generation of devs step down. I'd feel more comfortable if we've let kernels stay minimal rather than becoming kitchen sinks. Apart from better portability, this would've also enabled younger devs to start from scratch rather than having to maintain daddy-o's monstrosity.
To your point, it seems like eventually this ends poorly for the incumbents, when they become too bloated and too difficult to extend or modify, since you have to work around some over-engineered high-availability process clustering to touch anything.
In Linux's case, I think the kernel itself is actually fine, but the user land is increasingly tightly glued to an increasingly convoluted set of RedHat stacks.
I mean, it always has -- RedHat made the modern posix threads library, Linux's PAM, and several other core parts, but at least those were pretty modular and "unixy", by comparison to the monolith being developed now.
Thank's god for that! An OS should build on its strenghts, not be a least common demoninator for portabillity's sake. If that was ok we might just as well just use one OS
This reduces the amount of code they are responsible for maintaining, and everyone involved gets the benefit of open source collaboration.
They have a blog post about it here:
https://www.redhat.com/en/blog/what-open-source-upstream
Its focuses on XFS from 1993.
Its Stratis idea is just a joke comparing to what ZFS is.
They should finally admin (like Canonical) that OpenZFS is the way of future filesystem for now.
Linux world should put their religion (GPL) aside here.
Besides above filesystem 'issue' the PulseAudio, systemd and SELinux are not hated without reason ... many people just 'love' what Lenny has to offer.