Years ago, freshly out of school, I ended up in a job at a french small shop that required me to learn a lot of things.
They had a brillant sysadmin, that I bothered to explain them to me. Sometimes, he'll come to my desk, looking at my screen, typing the solution while explaining the problem to me with simple words.
Standing up, typing with one hand, and working on what was supposed to be my specialty, he was doing a better job than I was, despite me interrupting his workflow.
I left the job, and levelled up. One day, I read an article about docker, and the name of the dev caught my eyes: it was the friendly former sysadmin colleague, Jérôme Petazzoni, who went on becomming docker's biggest evangelist.
Thank you Jérôme, for making sure I started my carrer by not being the smartest guy in the room. I kept it as a habit.
> for making sure I started my carrer by not being the smartest guy in the room
That's indeed invaluable. That should be the first advice to anyone starting out: if you find yourself being the smartest in a room, go find a new room. Right now.
I left the company I was part-timing at because after college I realized I was the smartest one in the room; my boss thought some of the code I wrote was "brilliant" and for this reason did not let me improve/fix any of it, which led to the company shipping a broken product to some very annoyed customers.
If you are ever managing someone who is "your best employee" and they want to do something you think is useless, please ask them why and what the expected results will be. Also when you get that data, listen a little? If priorities clash do a balancing act: "If you can do X in Y time, do it. But if day D comes along, switch over to my priority".
I graduated college and started working at a local tractor manufacturer. In hindsight, this company had no idea how to make software.
For instance, they required that all "software engineers" were from an accredited engineering program--at the time, none of the universities in the area had an accredited software engineering program, so they would hire electrical engineering majors who had no desire to write software but were looking for a way to get their foot in the door. Most of the EE majors had a single semester of C and that was their entire software qualification; however, I double majored in EE and CS and my passion was software. That said, when I graduated I didn't understand any of this--I just figured there was this successful company and there was so much I didn't know about software, so I thought I would learn loads there--and I did learn loads, but it was mostly about what not to do.
One of my first projects was to take over an embedded C++/Qt project that had been bid out to some contractors in the Ukraine. The software was super buggy and the code was immensely convoluted. Test coverage was part of the contract, so it had tests, but even still, no one could change anything without breaking everything. It took me weeks of plotting the code paths to figure out that there were many paths through the code that were only exercised by tests--literally they were just trying to boost their code coverage metrics by adding code that was unreachable at production. I guess it didn't occur to me that professional software engineers would be so bad at writing code, and that a large and well-known software company would hire them.
Thankfully, after eventually gaining my entirely-nontechnical manager's trust, we were able to bring in a consultancy who helped me demonstrate that it would be no more than half the cost to rewrite the code rather than try to fix all of the existing bugs even if we never found other bugs or introduced a single bug of our own. We did the rewrite and it was successful, but I was beginning to realize that in some overly simplistic sense, I was the "smartest" one in the room as it pertained to software development anyway.
Ten years later, when I talk to my peers who stayed, many of them have been writing embedded C programs and various MatLab models and they still don't understand pointers or dynamic memory or git or anything outside of their very local niche. Which would be fine if they were happy, but now many want to move to another company but they lack the skills and confidence to get anything more than an entry level job at many other software companies--so much of their compensation is derived from their knowledge of their employer's business and processes rather than their technical acumen, so they would likely have to take a significant pay cut to move to another employer.
Anyway, this is already a very long post despite oversimplifying many things, but I hope this illustrates what it looks like to be the smartest person in the room and why it's not good.
When I read people complaining about "Small Images" like mentioned here where the author seems to warn against it and even proposes making sure enough "tooling" is in the container.
I'm always a bit curious why more people don't talk about using nsenter in the rare situation they need to inspect stuff in a running container?
For a production microservice container, why have a bunch of bloat in the container that increases the surface area to manage while also only actually being used in edge cases?
How do you use nsenter efficiently for running more than one program? You might need dozens of tools (and all their dependencies) to diagnose and fix issues.
I aim for "small images" because downloading a full install of Ubuntu is rediculous on anything but gigabit fibre.
But I don't aim for minimal images. Using alpine and installing what ever you like without worrying about all the bullshit to make caches stay empty still results in images only a few megs, don't need to do much better than that.
Though I tend to agree, if you're talking about a compiled language. Static binary and there is little need for anything other than `from scratch`.
I shall look into nsenter. I have never actually used it. It may cover this use case.
As for the status quo, I spend a lot of time extracting dumps from things in containers and debugging weird issues which is really really difficult without the ability to install any tools on the container at will. When done, we blast the container and let kubernetes reschedule a new one, disposing of all tools and state.
On bloat, mine are sitting on top of a debian container layer, mostly because it’s a hell of a lot easier dealing with some dynamic libraries than fecking around with static linking stuff and copying dependencies into a minimalist container.
Edit: There is overlap between these two things which nsenter may not be able to solve looking at what I am reading.
Minimal containers shouldn't need that though. Your logs should tell you enough about the application it's running. K8s (get/describe) should tell you about the lifecycle of the pod. Lastly something like "dive" should tell you all you need about the image's file layout.
You can run a small sidecar with a shell and if you enable shared process namespace you can access the filesystem of the main "distroless" container via /proc (e.g. /proc/123/root)
I think the difficulty arises in the fact that you can't spawn the sidecar without terminating a pod, as far as I know. So you either create a sidecar for every pod, "just in case", or you have to restart your pod for debugging purposes (and modify your YAML files).
This is a great article, I just have one tiny nitpick: please add the publication date!
Information like this is very dependent on the context of when it was written, and something that is "best practice" or "an anti-pattern" will not always be that way. Adding a publication date and/or a "last updated" date to the top of the article is very helpful for readers to align the information in their minds.
EDIT: I realized just after posting this comment that the publication date is in the url. Time for that second cup of coffee :)
The small image problem seems like it may be best solved through volumes.
Consider that busybox is just under 1MB when dynamically linked and installed. That's roughly 50% larger than a distroless container.
Further, distroless doesn't just target size for size's sake - it's also to ensure that an attacker in the container doesn't have tooling that they like to use.
Instead if your containers are mounting a volume, say `/var/run/debug-tools/`, you could copy the tools into that mount before entering your containers. This probably has a good dev UX in that you can enter containers and have your tools, it amortizes the cost across your containers on a given host, and it isn't container specific ie: you perform one copy and now all containers on that host have it.
RE small images: advice like this always seems to ignore that Docker images are cached incrementally. If you build, say, a Go binary and then put it in a popular stock Debian or Alpine Docker image, pushing the final image to a registry will only transfer the diff over the network. The size of the diff is basically the compressed Go binary and doesn’t depend on the base image you’re using!
Then when you pull the Docker image onto a machine it’s likely the base image is already in cache so again it’s only the diff that is transferred.
It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization.
> It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization.
Not always. If you have thousands of containers running, the network cost* for changing the base image is massive because each instance needs possibly all new layers, not just the last layer or 2. Having a smaller base image is very practical at this point.
* And for network cost, there is both the literal cost with egress from dockerhub, or another registry, but also the load on your network infra.
Actually using that caching in a way that benefits you is, well, very hard. Especially since a lot of the time you have a lot of stuff included in order to start the final program, and every step above the base image will result in different layer that will be separate and probably not cacheable
Also, Alpine images are used for being, well, small in general - there's not much to exploit on such image, nor is there going to be a lot of unplanned-for data, and especially it will speed up deployment when you upload a new image.
It takes special, dedicated CI/CD work to have common base images to exploit sharing of layers properly.
While that's true, you or your company have to standardize on a base image to get benefit of shared image layers.
In my job, I rarely have an exact match on the base layer because I'll use images from Docker Hub. Although many of those images use the same small set of distros (Alpine, Debian, or Ubuntu), any given image is likely from a different base version of that distro, thus there is no common base image.
There may be differences in the base image even if you use the same tag since the same image may change over time. Thus if you pull debian:bullseye today, and then pull again tomorrow, you may have two different images.
Generally I don't bother trying to get a common base image and I'm happy enough just attempt to lower the overall size of my images.
> It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization.
I do use Debian Slim over Alpine without thinking but image size does matter in a lot of cases once you leave your dev box or start to use Kubernetes.
For example if you're using AWS Fargate images aren't cached[0]. That means every time you deploy your app a new pod is going to get created and it has to wait for your image to download.
Fargate already has a 30-40ish second spin up time inherit to the platform. Having to wait X seconds or even minutes more to download your app's image delays things quite a lot, especially if you have something like a pre-sync hook running to do something like running a database migration. Now every deploy pays this penalty twice.
While I won't address each item in your list. I'll say that I love Dockerfiles.
They make it clear what steps occurred to arrive at the resulting image. For any image that is built from an open-source repository, I can find the Dockerfile, look at it and say "Oh, that's how they did it!"
In the article it says that Dockerfiles might not be the best way to build your image. I think some of that may have come from before the days of multi-stage builds. With multi-stage builds you can use a lot of tools (like the Go compiler) at build time without including those tools in your final build.
Can we do better than Dockerfiles as they are today? I'm sure we can, but can I appreciate what I have while knowing it could be better.
Docker containers are easy to build in a way that's pretty bad. I don't think that necessarily implicates Docker containers as a bad tech. In the same way, you can write an insert language here application that's horribly unmaintainable, and that tells you nothing of the language.
I don't like Docker as a company, but I can't imagine living my tech life without containers + Kubernetes, and Dockerfiles are really easy to write and read.
I would add that using Docker in the first place is an anti-pattern. If you're using software you wrote yourself that can be made self-contained, e.g. if it's written in Go or Rust, then you probably don't even need Docker.
People always say "but how can you isolate the network?" and use Kurbernetes and .. and sure Docker is the easiest way to do that. But in my experience people rarely use Docker for that feature anyway. 99% of Docker use is because most Linux software is written in such a way that it is an absolute pain in the arse to distribute so shipping a whole OS is the bodgy solution that everyone accepts.
> Or, if you use Docker, you can use docker cp to copy some static tools (e.g. busybox) to a running container and check what’s going on.
I tried this recently with `kubectl cp` for my scratch image and it turns out that command will fail if the target container lacks a `tar` program in its PATH. I guess under the covers it's just doing a `kubectl exec <pod> -- /bin/tar ...` on the file rather than asking the kubelet to untar the file into the target container's filesystem.
33 comments
[ 323 ms ] story [ 2769 ms ] threadThey had a brillant sysadmin, that I bothered to explain them to me. Sometimes, he'll come to my desk, looking at my screen, typing the solution while explaining the problem to me with simple words.
Standing up, typing with one hand, and working on what was supposed to be my specialty, he was doing a better job than I was, despite me interrupting his workflow.
I left the job, and levelled up. One day, I read an article about docker, and the name of the dev caught my eyes: it was the friendly former sysadmin colleague, Jérôme Petazzoni, who went on becomming docker's biggest evangelist.
Thank you Jérôme, for making sure I started my carrer by not being the smartest guy in the room. I kept it as a habit.
Or I'm just dumb :)
That's indeed invaluable. That should be the first advice to anyone starting out: if you find yourself being the smartest in a room, go find a new room. Right now.
If you are ever managing someone who is "your best employee" and they want to do something you think is useless, please ask them why and what the expected results will be. Also when you get that data, listen a little? If priorities clash do a balancing act: "If you can do X in Y time, do it. But if day D comes along, switch over to my priority".
For instance, they required that all "software engineers" were from an accredited engineering program--at the time, none of the universities in the area had an accredited software engineering program, so they would hire electrical engineering majors who had no desire to write software but were looking for a way to get their foot in the door. Most of the EE majors had a single semester of C and that was their entire software qualification; however, I double majored in EE and CS and my passion was software. That said, when I graduated I didn't understand any of this--I just figured there was this successful company and there was so much I didn't know about software, so I thought I would learn loads there--and I did learn loads, but it was mostly about what not to do.
One of my first projects was to take over an embedded C++/Qt project that had been bid out to some contractors in the Ukraine. The software was super buggy and the code was immensely convoluted. Test coverage was part of the contract, so it had tests, but even still, no one could change anything without breaking everything. It took me weeks of plotting the code paths to figure out that there were many paths through the code that were only exercised by tests--literally they were just trying to boost their code coverage metrics by adding code that was unreachable at production. I guess it didn't occur to me that professional software engineers would be so bad at writing code, and that a large and well-known software company would hire them.
Thankfully, after eventually gaining my entirely-nontechnical manager's trust, we were able to bring in a consultancy who helped me demonstrate that it would be no more than half the cost to rewrite the code rather than try to fix all of the existing bugs even if we never found other bugs or introduced a single bug of our own. We did the rewrite and it was successful, but I was beginning to realize that in some overly simplistic sense, I was the "smartest" one in the room as it pertained to software development anyway.
Ten years later, when I talk to my peers who stayed, many of them have been writing embedded C programs and various MatLab models and they still don't understand pointers or dynamic memory or git or anything outside of their very local niche. Which would be fine if they were happy, but now many want to move to another company but they lack the skills and confidence to get anything more than an entry level job at many other software companies--so much of their compensation is derived from their knowledge of their employer's business and processes rather than their technical acumen, so they would likely have to take a significant pay cut to move to another employer.
Anyway, this is already a very long post despite oversimplifying many things, but I hope this illustrates what it looks like to be the smartest person in the room and why it's not good.
I'm always a bit curious why more people don't talk about using nsenter in the rare situation they need to inspect stuff in a running container?
For a production microservice container, why have a bunch of bloat in the container that increases the surface area to manage while also only actually being used in edge cases?
But I don't aim for minimal images. Using alpine and installing what ever you like without worrying about all the bullshit to make caches stay empty still results in images only a few megs, don't need to do much better than that.
Though I tend to agree, if you're talking about a compiled language. Static binary and there is little need for anything other than `from scratch`.
As for the status quo, I spend a lot of time extracting dumps from things in containers and debugging weird issues which is really really difficult without the ability to install any tools on the container at will. When done, we blast the container and let kubernetes reschedule a new one, disposing of all tools and state.
On bloat, mine are sitting on top of a debian container layer, mostly because it’s a hell of a lot easier dealing with some dynamic libraries than fecking around with static linking stuff and copying dependencies into a minimalist container.
Edit: There is overlap between these two things which nsenter may not be able to solve looking at what I am reading.
In theory. In practice this is only rarely the case.
† https://kubernetes.io/docs/tasks/debug-application-cluster/d...
That is, if we're talking Kubernetes of course.
Otherwise you have to prepare in advance and add a small sidecar to all pods that need it
Information like this is very dependent on the context of when it was written, and something that is "best practice" or "an anti-pattern" will not always be that way. Adding a publication date and/or a "last updated" date to the top of the article is very helpful for readers to align the information in their minds.
EDIT: I realized just after posting this comment that the publication date is in the url. Time for that second cup of coffee :)
Consider that busybox is just under 1MB when dynamically linked and installed. That's roughly 50% larger than a distroless container.
Further, distroless doesn't just target size for size's sake - it's also to ensure that an attacker in the container doesn't have tooling that they like to use.
Instead if your containers are mounting a volume, say `/var/run/debug-tools/`, you could copy the tools into that mount before entering your containers. This probably has a good dev UX in that you can enter containers and have your tools, it amortizes the cost across your containers on a given host, and it isn't container specific ie: you perform one copy and now all containers on that host have it.
Then when you pull the Docker image onto a machine it’s likely the base image is already in cache so again it’s only the diff that is transferred.
It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization.
Not always. If you have thousands of containers running, the network cost* for changing the base image is massive because each instance needs possibly all new layers, not just the last layer or 2. Having a smaller base image is very practical at this point.
* And for network cost, there is both the literal cost with egress from dockerhub, or another registry, but also the load on your network infra.
Also, Alpine images are used for being, well, small in general - there's not much to exploit on such image, nor is there going to be a lot of unplanned-for data, and especially it will speed up deployment when you upload a new image.
It takes special, dedicated CI/CD work to have common base images to exploit sharing of layers properly.
In my job, I rarely have an exact match on the base layer because I'll use images from Docker Hub. Although many of those images use the same small set of distros (Alpine, Debian, or Ubuntu), any given image is likely from a different base version of that distro, thus there is no common base image.
There may be differences in the base image even if you use the same tag since the same image may change over time. Thus if you pull debian:bullseye today, and then pull again tomorrow, you may have two different images.
Generally I don't bother trying to get a common base image and I'm happy enough just attempt to lower the overall size of my images.
I do use Debian Slim over Alpine without thinking but image size does matter in a lot of cases once you leave your dev box or start to use Kubernetes.
For example if you're using AWS Fargate images aren't cached[0]. That means every time you deploy your app a new pod is going to get created and it has to wait for your image to download.
Fargate already has a 30-40ish second spin up time inherit to the platform. Having to wait X seconds or even minutes more to download your app's image delays things quite a lot, especially if you have something like a pre-sync hook running to do something like running a database migration. Now every deploy pays this penalty twice.
[0]: https://github.com/aws/containers-roadmap/issues/696
> Big images
> Requiring rebuilds for every single change
> Forcing things to run in containers
> Using overly complex tools
> Building with Dockerfiles
They make it clear what steps occurred to arrive at the resulting image. For any image that is built from an open-source repository, I can find the Dockerfile, look at it and say "Oh, that's how they did it!"
In the article it says that Dockerfiles might not be the best way to build your image. I think some of that may have come from before the days of multi-stage builds. With multi-stage builds you can use a lot of tools (like the Go compiler) at build time without including those tools in your final build.
Can we do better than Dockerfiles as they are today? I'm sure we can, but can I appreciate what I have while knowing it could be better.
I don't like Docker as a company, but I can't imagine living my tech life without containers + Kubernetes, and Dockerfiles are really easy to write and read.
Unfortunately avoiding the anti-patterns seems to be much harder than adopting them.
I would add that using Docker in the first place is an anti-pattern. If you're using software you wrote yourself that can be made self-contained, e.g. if it's written in Go or Rust, then you probably don't even need Docker.
People always say "but how can you isolate the network?" and use Kurbernetes and .. and sure Docker is the easiest way to do that. But in my experience people rarely use Docker for that feature anyway. 99% of Docker use is because most Linux software is written in such a way that it is an absolute pain in the arse to distribute so shipping a whole OS is the bodgy solution that everyone accepts.
I tried this recently with `kubectl cp` for my scratch image and it turns out that command will fail if the target container lacks a `tar` program in its PATH. I guess under the covers it's just doing a `kubectl exec <pod> -- /bin/tar ...` on the file rather than asking the kubelet to untar the file into the target container's filesystem.