Ask HN: What is the best source to learn Docker in 2023?
I know docker for a long time but I never used it in serious projects. I find it hard to find good and especially up to date tutorials. I’m especially interested in a tutorial on how to run (feature branch) containers which spin up on bitbucket pipelines/GitHub actions for manual and automated testing for PHP and Node applications (containers should exist as long as a feature branch exists). I found some random YouTubers which give a nice intro/demo into some features but not really a deep dive.
83 comments
[ 2.9 ms ] story [ 150 ms ] threadI think this statement is not as precise as it should be. I'm running docker produced images in a k8s cluster and had to google what you are talking about here.
https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-...
"Docker-produced images will continue to work in your cluster with all runtimes, as they always have."
Dockerfile is a very common file to see in projects, and I'm thankful when I see it.
Although a pattern I commonly see these days with a lot of OSS projects is that oftentimes they provide a docker-compose.yaml even though no one at any reasonable scale is running Docker Compose in production. This is simply because Kubernetes setups are complicated, weighty, and heterogenous, and often not running on your local machine and Docker Compose is a great way to do a hello world style demo of your container-orchestrated app (because everyone still uses Docker Desktop).
Not sure about this, Docker is still widely used in CI systems and local development in my experience.
It's also widely used in Opensource projects for "reproducible builds"
that's just my guess though :) Happy hacking!
How did you gather this? OP only mentions CI/testing, nothing about deployment. That's a long distance from needing kubernetes.
btw, imo, nearly all commercial shops' use cases are a long distance from needing kubernetes, but that doesn't seem to be stopping very many people... :)
Start step by step.
Before building on Github Actions, build locally.
See if you can build and tag and image with the git SHA. Then run your automated test command against the image/container.
Then see if you can write a github action doing exactly what you did locally.
Random blog posts have been more helpful in my experience vs youtube videos.
1) your reverse proxy, Nginx/Caddy
2) your "app", or API, whatever. pick whatever you want, a Rails API, a Phoenix microservice, a Django monolithic app, whatever you want.
3) your database. Postgres, whatever
4) Redis - not just for caching. Can use it for anything that requires some level of persistence, or any message bus needs. They even have some plugins you can use (iirc, limited to enterprise plans... maybe?) like RedisGraph.
5) elasticsearch, if you need real-time indexing and search capabilities. Alternatively you can just spin up a dedicated API that leverages full text search for your database container from 3)
6) ??? (sky is the limit!)
I prefer docker compose to Kubernetes because I am not a megacorp. You just define your different services, let them run, expose the right ports, and then things should "just work"
Sometimes you need to specifically name your containers (like naming redis container `redis`, and then in your code you will have to use `redis` as the hostname instead of `localhost` for example).
basically That's It (tm)
In which cases would you prefer Kubernetes? Or rather, why would a megacorp prefer it over standard Docker?
"Docker compose" (not the same thing as docker) works great at single machine scale/local development environments, but isn't really designed to scale much beyond this to production environments, multiple servers and data centers etc, which Kubernetes is. This isn't to say you couldn't deploy something to production with compose, its just not very likely outside of small personal projects - there are heaps of features in Kubernetes that simply don't exist in compose.
Generally you'd typically find a docker compose configuration for easy local development environment deployments, a Kubernetes configuration for managing the production environments, although there are no hard and fast rules here. Compose generally works best where the services fit all on the same box, which is rare for a business of almost any size in production, but common for local dev work.
I also prefer Compose for personal projects and local development, but it simply wouldn't work at any place I've worked for production deployments.
[1] https://minikube.sigs.k8s.io/docs/start/
My workflow typically looks like this:
1. Run app locally during dev without Docker
2. Build and run Docker image locally with Docker compose
3. Deploy to development Kubernetes cluster
4. Deploy to production Kubernetes cluster
When you have many developers, the cost for maintaining one dev cluster per developer quickly goes up. One cluster for all developers can be used for testing/staging, but not for development.
Minikube is a replacement for your step (2), except you can now use your existing tools (kubectl etc) instead of docker-compose.
In short:
I'd say that going from Docker Compose to Docker Swarm is the first logical step, because it's included in a Docker install and also uses the same Compose format (with more parameters, such as deployment constraints, like which node hostname or tag you want a certain container to be scheduled on): https://docs.docker.com/compose/compose-file/compose-file-v3... That said, you won't see lots of Docker Swarm professionally anymore - it's just the way the job market is, despite it being completely sufficient for many smaller projects out there, I'm running it in prod successfully so far and it's great.Another reasonably lightweight alternative would be Hashicorp Nomad, because it's free, simple to deploy and their HCL format isn't too bad either, as long as you keep things simple, in addition to them supporting more than just container workloads: https://www.hashicorp.com/products/nomad That said, if you don't buy into HashiStack too much, then there won't be too much benefit from learning HCL and translating the contents of various example docker-compose.yml files that you see in a variety of repos out there, although their other tools are nice - for example, Consul (a service mesh). This is a nice but also a bit niche option.
Lastly, there is Kubernetes. It's complicated, even more so when you get into solutions like Istio, typically eats up lots of resources, can be difficult to manage and debug, but does pretty much anything that you might need, as long as you have either enough people to administer it, or a wallet that's thick enough for you to pay one of the cloud vendors to do it for you. Personally, I'd look into the lightweight clusters at first, like k0s, MicroK8s, or perhaps the K3s project in particular: https://k3s.io/
I'd also suggest that if you get this far, don't be afraid to look into options for dashboards and web based UIs to make exploring things easier:
In the terminal, there are also a few useful projects:Kubernetes scales down pretty well, there's a few canned single-machine versions. I've been playing with k3s lately at work (if this works out, it should lead to things running on a proper HA cluster), and I'll probably also move over some of the standalone containers I have at home (which are all single-instance standalone things, currently using podman as a systemd service).
[0] https://www.udemy.com/user/bretfisher/
Once you have that under your belt it's not hard to work out how Docker itself works and how you can use it to fulfill the sort of CI/CD objectives you have outlined. Docker itself isn't important, the semantics of containerization are.
Something that Docker (and Docker like things) take massive advantage of are overly filesystems like AUFS and overlayfs, you would do good to understand these (atleast skin deep).
Finally networking becomes really important when you start playing with network namespaces, you should be somewhat familiar with atleast the Linux bridge infrastructure and how Linux routing works.
Good luck!
It's the most roundabout way - and OP is conflating docker with CI/CD, referencing PHP and Node - it's probably safe to say they aren't looking for a deep dive.
Plus - knowing how it runs under the hood doesn't mean you know how to use docker itself.
I think it's safe to say they do want a deep dive but might be forgetting to mention some of their reasons.
For someone familiar with docker, maybe it is good to start from the other side and work backwards.
You would think that is would still be advantageous to have a detailed understanding of what is going on in the stack, but that actually causes problems when you make a suggestion that no one else understands.
And this would be a great advice!
Docker implemented in around 100 lines of bash: https://github.com/p8952/bocker
This is the most mindblowing example for enterprise security teams that think Docker is a new threat on a single tenant Linux host.
No, buddies, all this stuff is already there. If you were fine with your visibility before*, you're still fine. Go find a real problem while people play with their developer dopamine.
* NARRATOR: They shouldn't have been.
Ask it something like “Explain how to get started with Docker” and it will give you a bunch of steps in a reasonable order. Then ask it for details for each step, like:
“How do I install Docker on macOS?”
“Write a commented Dockerfile for an application written in $WHATEVER”
“Now write a commented Docker Compose file for this application and a Postgres database”
etc
What does ChatGPT say for something like
"How do I clean up old docker images in a registry after I merge a branch?"
1. Rebuilding the entire container which often involves stopping and starting it, etc.
2. Manually running commands that copy the files into the container. This is irritating because if I forget which files I changed or forget to run the copy command I end up with a "half updated" container.
3. SSHing into the container. This is irritating because I have to modify the port layout and permissions of the container and later remember to "restore" them when I'm "done" making the container.
Thanks!
This works, however, only if you know beforehand which files you’re going to update frequently.
The data you work with should be stored externally (volume, database, accessed via API, …). You don‘t keep persistent state of your workload in the container.
Of course, this means that you'll not just stop, but completely destroy the old container and start a new one (created from the new image). You can get this to happen without service downtime by using the very same techniques that you'd use on any high availability / multi-server enviornment (rolling upgrades, canary deployments, etc.).
If you need to have some files that persist across these upgrades, then you use volumes and/or bind mounts. These allow you to have folders that persist independently of the container's lifecycle. They are typically used to store things like a sqlite database that the container uses, the set of configuration files that you can edit on a per-instance basis, etc.
Finally, there's a big case where you ignore all of the above: when you use containers as a development tool. In that case, particuarly for "interpreted" languages (python, php, ruby, etc.) it becomes extremely useful to bind-mount your pograms' sources inside a development container. You can then develop normally but also change the entire system where your app runs extremely easily. You can also keep different environments (language version, libraries, configuration of all those) for different projects without any change for conflicts between them, etc.
For development, you add a volume to the container and mount a location on that volume to a location on your host hard drive. There are different mount options that control which changes propagate in which direction, but two-way propagation is avaliable and works fine.
Once you have this set up, any changes you make on your host system are automatically reflected in your containers volume. It's very low stress and low overhead, honestly.
It demystified a lot of docker features for me.
This is the reason I gave up on learning docker properly. I had 3 devices at my disposal - M1 mac, a windows 10 pc and a rpi. The random errors I was getting made me quite frustrated. Keep a code diary and document your mistakes and solutions.
Also get a VPS. Never ever try a serverless solution when trying to properly learn docker. Also, do not try to do anything that involves GPU processing.
It all boil downs to that step 0, makes sure you have that "docker compatible" device. People are successfully running GPU processes using docker. However, docker is not "run everything, everywhere" as some people may think it is.
Really understanding the idea of containerization is fundamental. If someone tries to dabble in GPU processing in their first or second week of learning docker, they will be surprised how difficult troubleshooting docker is.
IMHO there’s still room for hiqh-quality blog posts about containers. E.g. there are lots of gotchas that could be explained. E.g. if you keep your commands in a suboptimal order, you will not get the benefit of caching when building the container. And why use multi-stage builds etc etc.
PS. See also https://xkcd.com/1053/ :)
For example. Docker has absolutely zero knowledge of the branches lifetime or even branches at all. This is something you have to design using the existing capabilities of docker together with features or existing integrations provided by GitHub or bitbucket.
Of course knowing docker deeper will help you understand these boundaries better and use them.
One secret is that there is actually not much to it, most things are just variations of docker run and various tricks within docker build, sprinkled with some volume and image management like tagging and pruning. Other orchestrators like GH Actions, Compose, Kubernetes etc can be seen as building around these basic blocks.
If you already know these basics, you are probably going to learn faster by getting your hands dirty, trying to solve the scenarios you need, rather than binge watching tutorial#187 on YouTube.
https://github.com/nickjj/docker-node-example is an up to date Node example app[0] that's ready to go for development and production and sets up GitHub Actions. Its readme links to a DockerCon talk from about a year ago that covers most of the patterns used in that project and if not some of my more recent blog posts cover the rest.
None of my posts cover feature branch deployments tho. That's a pretty different set of topics mostly unrelated to learning Docker. Implementing this also greatly depends on how you plan to deploy Docker. For example, are you using Docker Compose or Kubernetes, etc..
[0]: You can replace "node" in the GitHub URL with flask, rails, django and phoenix for other example apps in other tech stacks.