+1 for using hadolint! I have been using it for ages, and I highly recommend it. I also sometimes write some Goss tests, and even run Trivy at the final stages of the container build and delete them once done in the same layer. It takes about 10 seconds or so longer to build the container, but its baked in and helps you catch issues before pushing the container.
> How does Docker handle asynchronous intermittent processes? Not good, actually. It is well known in Docker 101 that thou shall not run cron in the same container as the main process.
You can run cron just fine in Docker containers. Whether or not that's a good idea depends on what the cron is doing, what your application is doing, your runtime requirements, etc. Running cron in your container is definitely way simpler than the other options involving container scheduling.
This. The advice to move cron jobs outside has its place, of course. But unless you’re running at massive scale, Docker becomes easier once you stop worrying about what is and isn’t “proper container architecture”, and just remember that it’s Linux.
I think "1 container = 1 process" is a common misconception. There should be a separation of concerns, but there's no reason to go to extremes where it doesn't make sense.
When docker first arrived, people were confused about how to use it. I remember seeing lots of people putting an ssh daemon in their containers!?! I like the s6-overlay approach, it's pragmatic.
I haven't been using containers as much lately, but I wonder if s6-overlay's approach could be used to justify including a database into the container with an application. Is that a good idea?
> I haven't been using containers as much lately, but I wonder if s6-overlay's approach could be used to justify including a database into the container with an application. Is that a good idea?
The answer is as always: it depends. My rule of thumb here would be: would it ever make sense to configure the application to use a different (or even just remote) database? If so, then the database should have a separate container (when using a local db). This applies most of the time.
Similarly, if the database itself is an integral part of the application which it doesn't make sense to swap out or have remote... then by all means just include it and tell the user to mount their data volume to /data or whatever. Example: A single-system file indexer.
>would it ever make sense to configure the application to use a different (or even just remote) database? If so, then the database should have a separate container (when using a local db).
This can probably be further simplified to "does it run over ports? then probably yes". E.g. you'd separate your application and database, but not your database and its filesystem.
I'm not sure I fully understand, but with my assumptions in hand:
That's bit of an implementation detail, I think.
If my app is the most amazing file system indexer using PostgreSQL behind the scenes, I don't think the "port" distinction is relevant... oh, wait. You're thinking of ports as in INET vs. plain sockets?
That took me a long time to get. A very technical way to put it, but thank you.
> would it ever make sense to configure the application to use a different (or even just remote) database?
Emh, don’t we always end up like this? As in, the database stays up while restarting the app; The DB is restarted when upgrading versions; and when we start 10 front-end nodes while keeping one DB only. All if this because they have different lifecycles: DB data is permanent, webserver data is ephemeral.
The reason for 1 container, 1 process depends upon architecting your application in such a way that it works with it. Ideally, each application thread is stateless and independent of all others, most or all of them don't require graceful shutdown, and you can encapsulate state elsewhere in persistent volumes. If you're just using a single-node container runtime engine, it's not a huge benefit, but it does make fearless restarts a lot simpler when all you have to do is stop and start the container. Think of Erlang style crash-only programming where crashing is the only way to stop a process.
When you get into multi-node orchestrators, though, then the benefit becomes much more apparent, especially at a scale where nodes are guaranteed to fail every now and again. Rescheduling to another node is then, again, as simple as launch a new container from the same image on a different node. That happens a lot faster if each container only needs to start one process.
It also enables you to practice chaos engineering, which is the practice of intentionally crashing your own containers every few hours. This is a good practice for security, because if an attacker ever gains a foothold in one of your containers, they'll quickly lose it, but also to just force your infrastructure people and application architects to design in such a way that you're guaranteed to be treating servers like cattle instead of pets, because you're constantly killing them and have no choice if you want your application to still work. This ensures what you're deploying is robust and repeatable and doesn't secretly depend on some specific server state an admin achieved at 1 in the morning some random Sunday hopped up on Red Bull that he'll never remember and never be able to recreate.
There's nothing intrinsic about crons in the same container that prevent you from rescheduling containers on other nodes, fearlessly restarting, or whacking random containers. What they're doing, how they're handling state, and their lifecycles might.
Study your requirements. Whether you run your crons in the same container or a separate container depends on your system's behavior and failure/recovery scenarios. There are complexity tradeoffs--moving the cron scheduling outside of the container suddenly means you have moving parts outside of your container to test, instrument, monitor, and maintain. If you're already doing that, it's not too big of a deal, but if you aren't then I would think twice before adding all this complexity to your architecture, and make sure it's really worth it.
(I can't tell from the article which approach works for their specific use cases...I'm pushing back against the general advice to "avoid cron in containers". I would default to using cron, and only move those processes to separate containers if you have a concrete reason to.)
I can see a PHP centric shop not feeling all the upside of containers. The typical PHP model is pretty simplistic, so it's not hard to replicate most of the docker benefits without using containers at all.
That is, I can manage and deploy php apps with different php versions, database versions, repo versions, etc, without too much pain already. Because deploying a PHP app is mostly just dropping files into a directory and restarting a web server.
How do you do that one? Last time I tried (granted, a long time ago), apache-mod-php was hardcoded into a single php version, and you couldn't have different versions of it on apache either.
I always classified php as one of the most problematic languages that need containment. There's not only that hard dependency on the version, but the language also has a global configuration storage that, last time I used it, can not be overloaded.
You use PHP FPM (Fast CGI) behind Nginx, so you run multiple PHP-FPMs each with their own PHP version. This falls out of stock Debian/Ubuntu already and can be easily extended to even most recent and older PHP versions through PHP Debian Maintainer Ondrejs PPA. Currently PHP 7.0 - 7.4 and 8.0 available via apt-get
There's a number of ways to deal with it now. Having different versions of php installed is the easy part, having the webserver send requests to these is a little more tricky. Using PHP-FPM makes this considerably easier as you can have different versions listening on different unix sockets, for example.
I seem to recall being able to do this with some effort in the old days. You might have to compile the mod_php's yourself though, looks like at least FreeBSD packages have different php versions conflict with each other.
If you use php_fpm as others suggest, it would likely be easier.
Either way, you need to have some way to indicate which version you want; it used to be common on shared hosting to have .php3 vs .php4 vs .php5 files to indicate which version you wanted; but I think modern PHP tends to have less code that doesn't work with newer versions.
And yet, when I criticized Docker in 2018, in an article much discussed here on Hacker News, many of the comments pointed out how much something like PHP would benefit.
raziel2p wrote:
"However, I think the author trivializes the amount of work required to make different types of Python/PHP/NodeJS/whatever apps all work in a consistent way through configuration management, saying "I can just write a bash script or makefile." or "just download roles from Ansible Galaxy". This is so painfully ignorant and irresponsible that I fail to take the article seriously as a whole."
I can see both sides. If your PHP app depends on non-core modules, or resources that might clash with more than one instance running (apcu comes to mind, or redis), or a complex database layer, etc...maybe docker would help.
Quite correct, both sides are clearly visible and there might be some point to using containers anyway.
But there's a bigger issue here. The comments the OP received, are heavily worded to the point that they explicitly state that the article made them angry.
Regardless of whether containers are suitable or not for task X or Y, it is very difficult to have a normal conversation, if one's comments can actually make people angry.
If you find yourself assuming someone else is motivated by fear, first ask yourself if they might be motivated by truth-seeking.
I recommend applying Bayes Rule:
If someone says they were abducted by aliens, first think about how likely that is to have happened. Then account for the probability that someone who says that it happened is confused or lying about that situation. Then maybe ask a clarifying question.
If someone says they were frustrated and confused by python packaging, first think about how likely that is to have happened. Then account for the probability that someone who says that it happened is confused or lying about that situation. Then maybe ask a clarifying question.
——————————
Personally, I think it is much more common for people to be confused about configuring and packaging software than it is for people to be totally clear yet also confused about whether or not they are confused.
It's almost as if tyingq and raziel2p are two different individuals with their own unique viewpoints and opinions... It's not a knock on you - it's just that I never understood comments like this.
We had some software break earlier this week because our Ansible role installs Python 3.8.10 as `python` but the system had 3.8.6 installed as `python3`. This meant that when we ran `/usr/bin/env python` it used the non-system python which wasn't picking up packages from /usr/lib/python3.8/dist-packages or wherever Ubuntu packages are installed.
The default `pip` and `pip3` commands, however, used the system python and did read those directories, meaning that it kept insisting the given package was installed. It was a frustrating amount of digging around that could have been avoided in myriad ways, one of which being containers (though for a 30-line python script, docker seems like overkill).
To avoid this problem I use virtualenv for each project _and_ I point to the pip binary inside that env (env/bin/pip or env/bin/python) in my ansible scripts or any other commands.
I settled on this after running into similar problems
Although if you have multiple projects using the same framework and libraries on the same server, using containers allows you to use opcache preloading.
Yes, I didn't claim it as an absolute. Lots of solutions can restart a web server or a PHP-fpm instance without trouble. Or use things other than containers to manage zero downtime.
Lots of solutions don't want to - that's fairly different than "can't". I dare say few companies need nigh-100% uptime, and it costs an incredible amount both in setup and in maintenance to go from "10s outage while updating" to "no downtime while updating".
What exactly do you mean? How do these solutions deal with servers unexpectedly going down? I worked on telecom where you need the service to be up 24/7 and we restarted all the services on deployment. In waves of course.
Depends on how you set it up, but with a lot of container orchestration these days, you create a new image then point your load balancer at that instead, giving you a clean blue/green deploy without downtime.
> “Whatever is the underlying executing system, Docker can run it with the exact same code, byte by byte.”
Having recently delved into containers, I don't like this quote. It's not wrong, but I think it's also misleading. In particular:
> Whatever is the underlying execution system
Docker can only execute containers on Linux. Indeed, Docker itself is essentially a UI and management tool for Linux kernel features (chroot, bind mounts, etc.). Whilst Docker provides software for other platforms, like macOS, those are just UIs bundled with a VM/hypervisor running Linux; and it's that Linux system which runs the containers.
This may seem pedantic, but I think it's important to understand (a) what's actually happening when we run a container, and (b) what we can/can't do with such tools. For example, I was confused why my containers couldn't run macOS binaries: the whole point of containers is that they run inside the host OS, unlike VMs which have their own OS. That's certainly true, but I didn't realise that when running Docker on macOS, the host is still a VM running Linux!
> Docker can run it
Again, pedantic but important: Docker is a UI and management tool for containers, which are ultimately run by an underlying Linux system (usually via `runc`, or something compatible). I find this important since "Docker" provides much more than just running containers, e.g. it has Dockerfiles, images, registries, etc.
If someone just wants to run a container, they don't need Docker; and I've found Docker to be a more complicated and convoluted way of running containers than, say, OCI tools.
For example, I recently wrote an AWS Lambda function which needs to run from a custom container (it bundles some tools which are larger than Lambda's 50MB code limit; whilst containers have a 10GB limit). I originally did this with Docker, which required:
- Building an 'image' containing the software (I actually did this with Nix, rather than `docker build`, for reasons I'll give below)
- "Loading" that image into Docker
- "Tagging" the image with the URL of an ECR 'image repository' (we could include this tag during building, but I find this way keeps more distinction between 'building' and 'deploying')
- Running `aws ecr get-login | docker login` to "log in" to Docker. This seems ludicrous to me; I hear it's something to do with dockerhub compatibility or somesuch; which is still silly, since we're not using that.
- Pushing to ECR using Docker
In contrast, I've now switched that project to use OCI images, which only requires the following:
- Building an 'image' containing the software. This is just a .tar.gz file, plus some .json files which specify the "EntryPoint", the SHA256 of the .tar.gz, etc. (all easily created with bash + tar + jq)
- Uploading the image to ECR. This can be done with `aws ecr` shell commands, but they're quite low-level (e.g. files are uploaded 20MB at a time, which needs a loop) so I did this in Python using boto3. Note that the 'tag' is still needed, but it's just an argument to the 'ecr.put_image' function.
> the exact same code, byte by byte
This is my main problem with the quote. Whilst it might theoretically be possible to use Docker "properly", it seems to actively encourage awful practices.
For example, to run "the exact same code, byte for byte" we would need to actually get those bytes in the first place. Whilst the underlying container is simply a directory (known as a "bundle", which `runc` actually executes), Docker abstracts over such bundles: first we tar them up into "layers", which we then list in a JSON file called an "image", which we then "push" to an "image repository". To run a container, we "pull" its image from a repository...
The maddening thing is that Docker should be a lot more portable than it is; the actual specs and architecture need something to provide the primitives, but it doesn't have to be runc + Linux namespaces. Sure, the MacOS doesn't have those primitives, but ex. FreeBSD and Illumos both do. And, in fairness, with runj (https://github.com/samuelkarp/runj#runj) we might finally be moving in that direction.
Docker has a lot of quirks and rough edges that may be attributed to its history. It was open-sourced rather hastily from dotCloud, and caught on largely due to its appeal as an easy-to-use, open, and complete package for containerisation, unlike other containerisation ecosystems that already existed. In some ways it's an example of a "worse is better" solution (which is not a bad thing). Some of the things you mention are just the typical sort of bad design decisions that get made without the benefit of hindsight. I'm sure we have all made similar missteps while designing systems. It's also true that some of these flaws made it more approachable, and actually helped with adoption. For example, a reproducible image build would be great, but it would not be as easy for a newbie to throw together as a quasi-shell-script Dockerfile that lets you run whatever non-reproducible commands you like. That less-opinionated disposition of Docker was part of what made it easy to pick up. In the near future we will probably see a new containerisation ecosystem rise with an overall better design and more disciplined approach. Docker will always have its place in history as the implementation that made containerisation mainstream.
54 comments
[ 3.7 ms ] story [ 110 ms ] threadYou can run cron just fine in Docker containers. Whether or not that's a good idea depends on what the cron is doing, what your application is doing, your runtime requirements, etc. Running cron in your container is definitely way simpler than the other options involving container scheduling.
But it is important to at least show a little humility and recognising what problems they are solving even.
Hum... It's some Linux that appears out of nowhere and may go away at any time. This is the reason for most those container best practices.
Anyway, this really doesn't translate to "don't", but be sure to keep the transiency in mind.
S6-overlay people lay it out pretty neat https://github.com/just-containers/s6-overlay#the-docker-way
I haven't been using containers as much lately, but I wonder if s6-overlay's approach could be used to justify including a database into the container with an application. Is that a good idea?
The answer is as always: it depends. My rule of thumb here would be: would it ever make sense to configure the application to use a different (or even just remote) database? If so, then the database should have a separate container (when using a local db). This applies most of the time.
Similarly, if the database itself is an integral part of the application which it doesn't make sense to swap out or have remote... then by all means just include it and tell the user to mount their data volume to /data or whatever. Example: A single-system file indexer.
This can probably be further simplified to "does it run over ports? then probably yes". E.g. you'd separate your application and database, but not your database and its filesystem.
That's bit of an implementation detail, I think.
If my app is the most amazing file system indexer using PostgreSQL behind the scenes, I don't think the "port" distinction is relevant... oh, wait. You're thinking of ports as in INET vs. plain sockets?
That took me a long time to get. A very technical way to put it, but thank you.
Emh, don’t we always end up like this? As in, the database stays up while restarting the app; The DB is restarted when upgrading versions; and when we start 10 front-end nodes while keeping one DB only. All if this because they have different lifecycles: DB data is permanent, webserver data is ephemeral.
When you get into multi-node orchestrators, though, then the benefit becomes much more apparent, especially at a scale where nodes are guaranteed to fail every now and again. Rescheduling to another node is then, again, as simple as launch a new container from the same image on a different node. That happens a lot faster if each container only needs to start one process.
It also enables you to practice chaos engineering, which is the practice of intentionally crashing your own containers every few hours. This is a good practice for security, because if an attacker ever gains a foothold in one of your containers, they'll quickly lose it, but also to just force your infrastructure people and application architects to design in such a way that you're guaranteed to be treating servers like cattle instead of pets, because you're constantly killing them and have no choice if you want your application to still work. This ensures what you're deploying is robust and repeatable and doesn't secretly depend on some specific server state an admin achieved at 1 in the morning some random Sunday hopped up on Red Bull that he'll never remember and never be able to recreate.
Study your requirements. Whether you run your crons in the same container or a separate container depends on your system's behavior and failure/recovery scenarios. There are complexity tradeoffs--moving the cron scheduling outside of the container suddenly means you have moving parts outside of your container to test, instrument, monitor, and maintain. If you're already doing that, it's not too big of a deal, but if you aren't then I would think twice before adding all this complexity to your architecture, and make sure it's really worth it.
(I can't tell from the article which approach works for their specific use cases...I'm pushing back against the general advice to "avoid cron in containers". I would default to using cron, and only move those processes to separate containers if you have a concrete reason to.)
That is, I can manage and deploy php apps with different php versions, database versions, repo versions, etc, without too much pain already. Because deploying a PHP app is mostly just dropping files into a directory and restarting a web server.
How do you do that one? Last time I tried (granted, a long time ago), apache-mod-php was hardcoded into a single php version, and you couldn't have different versions of it on apache either.
I always classified php as one of the most problematic languages that need containment. There's not only that hard dependency on the version, but the language also has a global configuration storage that, last time I used it, can not be overloaded.
If you use php_fpm as others suggest, it would likely be easier.
Either way, you need to have some way to indicate which version you want; it used to be common on shared hosting to have .php3 vs .php4 vs .php5 files to indicate which version you wanted; but I think modern PHP tends to have less code that doesn't work with newer versions.
raziel2p wrote:
"However, I think the author trivializes the amount of work required to make different types of Python/PHP/NodeJS/whatever apps all work in a consistent way through configuration management, saying "I can just write a bash script or makefile." or "just download roles from Ansible Galaxy". This is so painfully ignorant and irresponsible that I fail to take the article seriously as a whole."
From here:
https://news.ycombinator.com/item?id=17062288
But there's a bigger issue here. The comments the OP received, are heavily worded to the point that they explicitly state that the article made them angry.
Regardless of whether containers are suitable or not for task X or Y, it is very difficult to have a normal conversation, if one's comments can actually make people angry.
If a person says that thing X is hard for them in their context, default to believing them.
There is no contradiction there.
I recommend applying Bayes Rule:
If someone says they were abducted by aliens, first think about how likely that is to have happened. Then account for the probability that someone who says that it happened is confused or lying about that situation. Then maybe ask a clarifying question.
If someone says they were frustrated and confused by python packaging, first think about how likely that is to have happened. Then account for the probability that someone who says that it happened is confused or lying about that situation. Then maybe ask a clarifying question.
——————————
Personally, I think it is much more common for people to be confused about configuring and packaging software than it is for people to be totally clear yet also confused about whether or not they are confused.
The default `pip` and `pip3` commands, however, used the system python and did read those directories, meaning that it kept insisting the given package was installed. It was a frustrating amount of digging around that could have been avoided in myriad ways, one of which being containers (though for a 30-line python script, docker seems like overkill).
Lots of solutions can't "just restart a web server" as part of their deployment.
You could always disable opcaching and then you wouldn't even have to restart the server to deploy a new PHP instance.
Having recently delved into containers, I don't like this quote. It's not wrong, but I think it's also misleading. In particular:
> Whatever is the underlying execution system
Docker can only execute containers on Linux. Indeed, Docker itself is essentially a UI and management tool for Linux kernel features (chroot, bind mounts, etc.). Whilst Docker provides software for other platforms, like macOS, those are just UIs bundled with a VM/hypervisor running Linux; and it's that Linux system which runs the containers.
This may seem pedantic, but I think it's important to understand (a) what's actually happening when we run a container, and (b) what we can/can't do with such tools. For example, I was confused why my containers couldn't run macOS binaries: the whole point of containers is that they run inside the host OS, unlike VMs which have their own OS. That's certainly true, but I didn't realise that when running Docker on macOS, the host is still a VM running Linux!
> Docker can run it
Again, pedantic but important: Docker is a UI and management tool for containers, which are ultimately run by an underlying Linux system (usually via `runc`, or something compatible). I find this important since "Docker" provides much more than just running containers, e.g. it has Dockerfiles, images, registries, etc.
If someone just wants to run a container, they don't need Docker; and I've found Docker to be a more complicated and convoluted way of running containers than, say, OCI tools.
For example, I recently wrote an AWS Lambda function which needs to run from a custom container (it bundles some tools which are larger than Lambda's 50MB code limit; whilst containers have a 10GB limit). I originally did this with Docker, which required:
- Building an 'image' containing the software (I actually did this with Nix, rather than `docker build`, for reasons I'll give below)
- "Loading" that image into Docker
- "Tagging" the image with the URL of an ECR 'image repository' (we could include this tag during building, but I find this way keeps more distinction between 'building' and 'deploying')
- Running `aws ecr get-login | docker login` to "log in" to Docker. This seems ludicrous to me; I hear it's something to do with dockerhub compatibility or somesuch; which is still silly, since we're not using that.
- Pushing to ECR using Docker
In contrast, I've now switched that project to use OCI images, which only requires the following:
- Building an 'image' containing the software. This is just a .tar.gz file, plus some .json files which specify the "EntryPoint", the SHA256 of the .tar.gz, etc. (all easily created with bash + tar + jq)
- Uploading the image to ECR. This can be done with `aws ecr` shell commands, but they're quite low-level (e.g. files are uploaded 20MB at a time, which needs a loop) so I did this in Python using boto3. Note that the 'tag' is still needed, but it's just an argument to the 'ecr.put_image' function.
> the exact same code, byte by byte
This is my main problem with the quote. Whilst it might theoretically be possible to use Docker "properly", it seems to actively encourage awful practices.
For example, to run "the exact same code, byte for byte" we would need to actually get those bytes in the first place. Whilst the underlying container is simply a directory (known as a "bundle", which `runc` actually executes), Docker abstracts over such bundles: first we tar them up into "layers", which we then list in a JSON file called an "image", which we then "push" to an "image repository". To run a container, we "pull" its image from a repository...
Saying "Docker can run" does not mean it will magically do so without work and preparation.
They are many ways indeed to take the same Dockerfile, build it 10 times and get 10 different Docker images.
People in the "infrastructure as code" ecosystem can relate, they faced the same challenges for years.
I don't care to migrate the infrastructure on which I depend to a proprietary product, especially one with a record of sudden, surprising changes.
It's perfectly possible to containerize without placing your faith in Docker.
Do your infrastructure work up front or later can both be a valid decision that can be different for each project depending on a lot of factors.