Most of my containers end up on k8s clusters as pods. What else would one use podman or docker for beyond local dev or maybe running a local containerized service?
Both podman and docker have pretty poor error handling in my experience.
It depends on the error, but for me it often comes down to a docker compose misconfiguration, resource, permissions, etc. In docker always find the errors quite difficult to trace back to root cause.
In podman you get a python a stack trace.
I wish both projects would assert different assumptions/requirements at runtime and report errors/warnings in a human-readable way.
The "podman generate systemd" command from the article is deprecated. The alternative are Podman Quadlets, which are similar to (docker-)compose.yaml, but defined in systemd unit files.
I would love to love Podman, but the fact that it regularly just fails to work on my Windows laptop (the WSL2 instance seems fine, but can't be connected to, the UI just says 'starting', and none of the menu options do anything) and that I can't figure out how to make IPv6 networking work on any platform means that Docker isn't going anywhere for the foreseeable future, I'm afraid...
Last third party container I’ve built had COPY —-link statements that didn’t worked on podman. Granted it worked just fine with a normal copy but it’s not 1:1.
> ... how to debug yet another layer of abstraction.
> Remove layers, keep things simple.
Due to the first line above, I'm not sure if I'm reading the second line correctly. But I'm going to assume that you're referring to the OCI image layers. I feel your pain. But honestly, I don't think that image layers are such a bad idea. It's just that the best practices for those layers are not well defined and some of the early tooling propagated some sub-optimal uses of those layers.
I'll just start with when you might find layers useful. Flatpak's sandboxing engine is bubblewrap (bwrap). It's also a container runtime that uses namespaces, cgroups and seccomp like OCI runtimes do. The difference is that it has more secure seccomp defaults and it doesn't use layers (though mounts are available). I have a tool that uses bwrap to create isolated build and packaging environments. It has a single root fs image (no layers). There are two annoyances with a single layer like this:
1. If you have separate environments for multiple applications/packages, you may want to share the base OS filesystem. You instead end up replicating the same file system redundantly.
2. If you want to collect the artifacts from each step (like source download, extract and build, 'make install', etc) into a separate directory/archive, you'll find yourself reaching out for layers.
I have implemented this and the solutions look almost identical to what OCI runtimes do with OCI image layers - use either overlayfs or btrfs/zfs subvolume mounts.
So if that's the case, then what's the problem with layers? Here are a few:
1. Some tools like the image builders that use Dockerfile/Containerfile create a separate layer for every operation. Some layers are empty (WORKDIR, CMD, etc). But others may contain the results of a single RUN command. This is very unnecessary and the work-arounds are inelegant. You'll need to use caches to remove temporary artifacts, and chain shell commands into a single RUN command (using semicolons).
2. You can't manage layers like files. The chain of layers are managed by manifests and the entire thing needs a protocol, servers and clients to transfer images around. (There are ways to archive them. But it's so hackish.)
So, here are some solutions/mitigations:
1. There are other build tools like buildah and packer that don't create additional layers unless specified. Buildah, a sister project of Podman, is a very interesting tool. It uses regular (shell) commands to build the image. However, those commands closely resemble the Dockerfile commands, making it easy to learn. Thus you can write a shell script to build an image instead of a Dockerfile. It won't create additional layers unless you specify. It also has some nifty features not found in Dockerfiles.
Newer Dockerfile builders (I think buildkit) have options to avoid creating additional layers. Another option is to use dedicated tools to inspect those layers and split/merge them on demand.
2. While a protocol and client/servers are rather inconvenient for lugging images around, they did make themselves useful in other ways too. Container registries these days don't host just images. They can host any OCI artifact. And you can practically pack any sort of data into such an artifact. They are also used for hosting/transferring a lot of other artifacts like helm charts, OPA policies, kubectl plugins, argo templates, etc.
> So any alternative tooling that forces Docker to get it's act together is welcome
What else do you consider as some bad/sub-optimal design choices of Docker? (including those already solved by podman)
I've wanted to migrate multiple times. Unfortunately, it failed on multiple places.
Firstly, podman had a much worse performance compared to docker on my small cloud vps. Can't really go into details though.
Secondly, the development ecosystem isn't really fully there yet. Many tools utilizing Docker via its socket, fail to work reliably with podman. Either because the API differs or because of permission limitations. Sure, the tools could probably work around those limitations, but they haven't and podman isn't a direct 1:1 drop in replacement.
I love podman, and, like others have said here, it does not always work with every container.
I often try to run something using podman, then find strange errors, then switch back to docker. Typically this is with some large container, like gitlab, which probably relies on the entirety of the history of docker and its quirks. When I build something myself, most of the time I can get it working under podman.
This situation where any random container does not work has forced me to spin up a VM under incus and run certain troublesome containers inside that. This isn't optimal, but keeps my sanity. I know incus now permits running docker containers and I wonder if you can swap in podman as a replacement. If I could run both at the same time, that would be magical and solve a lot of problems.
There definitely is no consistency regarding GPU access in the podman and docker commands and that is frustrating.
But, all in all, I would say I do prefer podman over docker and this article is worth reading. Rootless is a big deal.
Could never get rootless to work properly. I wanted to like Podman, but every time I wanted to use it there was some bump in the road that made me give up and use Docker.
To provide 1 contrary opinion to all the others saying they have a problem:
Podman rocks for me!
I find docker hard to use and full of pitfalls and podman isn't any worse. On the plus side, any company I work for doesn't have to worry about licences. Win win!
It's works great until you need that one option from Docker Compose that is missing in Podman Compose (which is written in Python for whatever reason, yeah...).
On a few machines now ive had Podmans windows uninstaller fail to remove all its components and cause errors on start up due to postman not being found. Even manually removing leftover services and start up items didn't fix the issue. Its a constant source of annoyance.
I write programs that run on the target OS again. It's much easier, turnaround time is much quicker, it's faster. Even battery lasts longer on my laptop. What the hell have we done to ourselves with these numerous layers of abstraction?!?
Podman networking is extremely unreliable. Our company made an effort to switch to get away from Docker Enterprise. We had to kill the effort because multiple people had random disconnects and packet drops with a range of services including K8S, Kafka, and even basic applications, both internal and in host network.
I tried podman for multiple times. Normal testing & sandox stuff just works and you really can do alias docker=podman. But ass soon as you add nertworking me broke for me. And for me it is really just a tool and I need my tools working. So I switched back.
Recently I did the GitLab Runner migration for a company and switched to rootless docker. Works perfectly, all devs did not notice all there runs now use rootless docker and buildkit for builds. All thanks to rootless kit. No podman problems, more secure and no workflow change needed
> If your Docker Compose workflow is overly complex, just convert it to Kubernetes YAML. We all use Kubernetes these days, so why even bother about this?
I find that kubernetes yaml are a lot more complex than docker compose. And while I do, no, not everybody uses kubernetes.
I disagree with you on that. Kubernetes YAML is on the same level of complexity as docker compose and sometimes even easier.
But verbosity - yeah, kubernetes is absolutely super-verbose. Some 100-line docker-compose could easily end up as 20 yamls of 50 lines each. kubectl really needs some sugar to convert yamls from simple form to verbose and back.
175 comments
[ 2.2 ms ] story [ 106 ms ] threadRemove layers, keep things simple.
That being said, it is here to stay. So any alternative tooling that forces Docker to get it's act together is welcome.
> Remove layers, keep things simple.
Due to the first line above, I'm not sure if I'm reading the second line correctly. But I'm going to assume that you're referring to the OCI image layers. I feel your pain. But honestly, I don't think that image layers are such a bad idea. It's just that the best practices for those layers are not well defined and some of the early tooling propagated some sub-optimal uses of those layers.
I'll just start with when you might find layers useful. Flatpak's sandboxing engine is bubblewrap (bwrap). It's also a container runtime that uses namespaces, cgroups and seccomp like OCI runtimes do. The difference is that it has more secure seccomp defaults and it doesn't use layers (though mounts are available). I have a tool that uses bwrap to create isolated build and packaging environments. It has a single root fs image (no layers). There are two annoyances with a single layer like this:
1. If you have separate environments for multiple applications/packages, you may want to share the base OS filesystem. You instead end up replicating the same file system redundantly.
2. If you want to collect the artifacts from each step (like source download, extract and build, 'make install', etc) into a separate directory/archive, you'll find yourself reaching out for layers.
I have implemented this and the solutions look almost identical to what OCI runtimes do with OCI image layers - use either overlayfs or btrfs/zfs subvolume mounts.
So if that's the case, then what's the problem with layers? Here are a few:
1. Some tools like the image builders that use Dockerfile/Containerfile create a separate layer for every operation. Some layers are empty (WORKDIR, CMD, etc). But others may contain the results of a single RUN command. This is very unnecessary and the work-arounds are inelegant. You'll need to use caches to remove temporary artifacts, and chain shell commands into a single RUN command (using semicolons).
2. You can't manage layers like files. The chain of layers are managed by manifests and the entire thing needs a protocol, servers and clients to transfer images around. (There are ways to archive them. But it's so hackish.)
So, here are some solutions/mitigations:
1. There are other build tools like buildah and packer that don't create additional layers unless specified. Buildah, a sister project of Podman, is a very interesting tool. It uses regular (shell) commands to build the image. However, those commands closely resemble the Dockerfile commands, making it easy to learn. Thus you can write a shell script to build an image instead of a Dockerfile. It won't create additional layers unless you specify. It also has some nifty features not found in Dockerfiles.
Newer Dockerfile builders (I think buildkit) have options to avoid creating additional layers. Another option is to use dedicated tools to inspect those layers and split/merge them on demand.
2. While a protocol and client/servers are rather inconvenient for lugging images around, they did make themselves useful in other ways too. Container registries these days don't host just images. They can host any OCI artifact. And you can practically pack any sort of data into such an artifact. They are also used for hosting/transferring a lot of other artifacts like helm charts, OPA policies, kubectl plugins, argo templates, etc.
> So any alternative tooling that forces Docker to get it's act together is welcome
What else do you consider as some bad/sub-optimal design choices of Docker? (including those already solved by podman)
Firstly, podman had a much worse performance compared to docker on my small cloud vps. Can't really go into details though.
Secondly, the development ecosystem isn't really fully there yet. Many tools utilizing Docker via its socket, fail to work reliably with podman. Either because the API differs or because of permission limitations. Sure, the tools could probably work around those limitations, but they haven't and podman isn't a direct 1:1 drop in replacement.
I know a lot of kubernetes fans migrate to podman, but if you use dev stacks.
Use in dev: devcontainer, podman can't replace docker!
I could reverse engineer all the cool user land stuff it does to make things seamless ... but who has the time ;-)
I often try to run something using podman, then find strange errors, then switch back to docker. Typically this is with some large container, like gitlab, which probably relies on the entirety of the history of docker and its quirks. When I build something myself, most of the time I can get it working under podman.
This situation where any random container does not work has forced me to spin up a VM under incus and run certain troublesome containers inside that. This isn't optimal, but keeps my sanity. I know incus now permits running docker containers and I wonder if you can swap in podman as a replacement. If I could run both at the same time, that would be magical and solve a lot of problems.
There definitely is no consistency regarding GPU access in the podman and docker commands and that is frustrating.
But, all in all, I would say I do prefer podman over docker and this article is worth reading. Rootless is a big deal.
Podman rocks for me!
I find docker hard to use and full of pitfalls and podman isn't any worse. On the plus side, any company I work for doesn't have to worry about licences. Win win!
I write programs that run on the target OS again. It's much easier, turnaround time is much quicker, it's faster. Even battery lasts longer on my laptop. What the hell have we done to ourselves with these numerous layers of abstraction?!?
```
> kubectl port-forward svc/argocd-server -n argocd 8080:443
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
E0815 09:12:51.276801 27142 portforward.go:413] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod 87b32b48e6c729565b35ea0cefe9e25d8f0211cbefc0b63579e87a759d14c375, uid : failed to execute portforward in network namespace "/var/run/netns/cni-719d3bfa-0220-e841-bd35-fe159b48f11c": failed to connect to localhost:8080 inside namespace "87b32b48e6c729565b35ea0cefe9e25d8f0211cbefc0b63579e87a759d14c375", IPv4: dial tcp4 127.0.0.1:8080: connect: connection refused IPv6 dial tcp6 [::1]:8080: connect: connection refused
error: lost connection to pod
```
People had other issues also. It looks nice and I would love to use it, but it just currently isn't mature/stable enough.
Recently I did the GitLab Runner migration for a company and switched to rootless docker. Works perfectly, all devs did not notice all there runs now use rootless docker and buildkit for builds. All thanks to rootless kit. No podman problems, more secure and no workflow change needed
I find that kubernetes yaml are a lot more complex than docker compose. And while I do, no, not everybody uses kubernetes.
But verbosity - yeah, kubernetes is absolutely super-verbose. Some 100-line docker-compose could easily end up as 20 yamls of 50 lines each. kubectl really needs some sugar to convert yamls from simple form to verbose and back.
Even if the tech is not top notch, Docker got a few things right on product management.