HN Explorer

Ask HN: What Happened to Docker 23?

3 points by bionsystem ↗ HN
Docker 23 installs by default now if you update your repository and run [apt|yum|dnf] [install|upgrade] docker-ce. However it breaks on builds (timeouts) and changes the location of config files (/etc/docker/daemon.json isn't used anymore), thus breaking our CI as well as making unusable our config management tools until we patch them.

The worst part is that the change occured on thursday, and there was no release notes when our CI started to break. So I was actually lucky to figure out that it was just that and forced version 20 everywhere before we would break everything. And today I found that some build timeout or fail in non-deterministic ways as well which seems to appear on some older versions as well according to google.

So I guess the lesson is, never use "latest" version of anything, anywhere, as they can't be trusted ? Has anybody encountered this kind of issues ?

Edit : I just found out that googling "docker 23 issues" also gives volume issues. This is all terrible for a major release that upgrades without warning.

8 comments

[ 7.3 ms ] story [ 32.5 ms ] thread
verdverm
As Kelsey Hightower once said "latest is not a version"

You should always pin versions to something specific.

I had to deal with new versions breaking things twice this week. One was the Docker Go client, which while pinned, updated to 23 and broke things. Had to roll it back, which goes to show you even pinning versions does not prevent these issues, though in your case, you would have known by being intentional. Of course, now you have the version update chores...

---

Docker changelog: https://docs.docker.com/engine/release-notes/23.0/ (published before the release, available for many weeks if you pay attention on GitHub)

bionsystem
The problem is that I don't control our production systems as they are on-prem on clients sites. Meaning that any of my client can run "apt update && apt upgrade" and I'm screwed regardless of what I do. This is in fact a good reason to keep "latest" on my side so that I know first-hand when things break.
bionsystem
The changelog was put online after the new version was released, too.
verdverm
The release notes were published on Feb 1, the assets on Feb 2
bionsystem
That seems to be true, I guess I checked on the wrong link on feb 2.
speedgoose
Yes it’s very classic to encounter breaking changes when you blindly install the latest version.
ezekg
Not to dismiss your hardship, but what was your expected behavior here? It installed the latest version for you, just as you asked... so I'm not sure how it can't be trusted lol
bionsystem
Gitlab and github both reports issues (excluding the configuration changes, we are talking timeouts during build and also issues with volumes apparently) that people experienced on day one. It's not like it's a hidden, complicated bug, if people like me experience it within minutes of the new version rolling out.

I guess we'll need to change strategy, maybe mark / fix every version everywhere and test both fixed versions and latest (in case our clients decide to upgrade on their own without consulting us first because that's still my main concern as mentionned in another comment), but that would double our CI costs and I'm not sure if I can find the time to duplicate every configuration.

I don't know maybe I'm too old for this.