Is it only me or does Docker change their mind about the upcoming policy every few days, and then present it in a way that blames the users for not understanding it?
> It would be great if some HNers on the anonymous and free tiers could go through the process and report on how throttling affected them in detail.
In my POV the first fundamental change that this caused was to cease considering docker hub as reliable infrastructure, and therefore started to look for solutions to reliably source container images. For instance, GitLab already provides its own container registry, and thus it makes sense to simply source images from them than from tertiary sources like Docker Hub.
Another impact that this has is that I'm looking into the need to operate a custom container registry for each Kubernetes cluster. It wouldn't be nice if an application running on Kubernetes or Docket Swarm risked downtime just because you happened to hit one of Docker Hub's rate limits.
> Well, the throttling seems to be on anonymous and free docker hub accounts for now. So you do have the option of paid tiers.
We don't need to resort to paid services. This is all about convenience.
As I've said, services like GitLab already provide container registry services, and we are also free to operate our own custom container registry services. Docker Hub is only used to source reference images such as Ubuntu, postgres, rabbitmq, traefik, etc etc etc. We can simply point to a different provider to continue accessing those. Even if these projects continue to use Docker Hub then if we stick with stable releases then we can simply point to a different container registry service, even one managed by us.
The main point is that Docker Hub ceased to be a reliable provider. Thus, we plan accordingly.
Exactly this. It‘s so easy to setup your registry using GitLab or then Docker Registry, Portainer and Traefik. No need to rely on external providers for a small team.
Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot - You would/should be caching most of the images from dockerhub anyways, so subsequential builds shouldn't count against your limit.
Internal repo that does caching of images is rather obvious and a good practice, but it still doesn't mean that 100 pulls are that many.
Maybe we understand something different by a "small" team, but IMHO a smaller team uses more ready public images, to save time and costs. Also even not that big project on Kubernetes that includes internal software will also have a lot of other software.
Let's speculate: I had a look at some teams I know and how they work (or they would like to work soon) and it's easily 15-30 images in a CI/CD pipeline. Plus tools used to develop, but not exactly released in a build (source code management, PM, communication etc). All teams I know have 2-3 cycles developed at all times (next release, hotfix, future release after the next one). This means a lot of these tools are multiplied by 2-3 (upgrading tools as well, so multiple versions of same tool needed). If you have a few clusters that have different requirements it can also be multiplied by 2 or 3.
I would not dare to think what would happen if someone used "latest" tag in their daily work, or organized a docker/kubernetes workshop for new people :)
> Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot
It really depends on your usecase. I mean, each CICD stage can involve building a brand new image and running a couple of them, sometimes more. 100 pulls per 6 hours corresponds to a constant rate of 3 min between pulls, which would require between 15 to 20 minutes for each commit.
Anyway, the primary practical implications of this sort of change is a) rate limiting suddenly becomes a requirement, b) we can no longer count on Docker Hub to provide a reliable service, thus we plan accordingly.
At one point, Docker's announcement said that all 'docker pull' requests would count towards the rate limit, even if you had every layer cached and no layers ended up being downloaded.
I may be wrong but I interpreted that to mean that even caching won't prevent you from hitting the limit.
If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Either you have way too many project for a small team to work on, or your CI configuration is incredibly wasteful.
Just a few years back I remember even the smallest Java shop would have a local "caching/forwarding" Maven repository and the upstream repos were only considered a fallback.
Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.
A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc.
I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.
The only thing that is being limited are pulls from the Docker Hub. If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
> The only thing that is being limited are pulls from the Docker Hub.
Yes.
> If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull.
Yes.
> Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?).
Not necessarily. You dockerize an app prior to running non-unit tests tests, and that image needs to be stored by a container registration service.
With this change, Docker Hub's ceases to be that service.
> You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
Nope, the point is that Docker Hub ceases to be relevant as a container registry. As long as you start to do all your work outside of Docker Hub, you won't need Docker Hub to do any of the work. You just get your base images from where you store your own container images and suddenly Docker Hub stops playing any role in your workflow, thus losing al relevance.
It looks like you don't, but I've always seen Docker Hub as...well...a hub, not just any ol' registry. It's where people publish final builds of the images that they release to the world and others pull them from there. If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling, you should be paying them for it or hosting your own registry.
If you're in a situation where having your own registry means Docker Hub "loses all relevance", then you're probably not the kind of user they have any reason to care about (and that's nonsense anyway, because you'll still almost certainly be using 3rd party images from there).
I'm sorry to say but your insistence of speaking out of a position of blisful ignorance and your refusal to open your eyes does not change the world around you.
> but I've always seen Docker Hub as...well...a hub, not just any ol' registry.
No one with the faintest cursory knowledge of Docker would ever claim that Docker Hub is anything other than a docker registry.
Docker itself advertises Docker Hub as a service "to create, manage, and deliver your teams' container applications." Docker itself describes Docker Hub as a "cloud-based application registry and development team collaboration services." That's their own official marketing slogan, taken straight out of their "Why Docker" page.
And even that you managed to miss and failed to understand.
> It's where people publish final builds
It really isn't. That's not how containers work at all, and that's not how Docker promotes the service, and that's not how anyone uses it. Nothing in your comments is even remotely compatible with basic notions of CICD.
Apparently you're so out of touch with containerization and CICD and devops workflow that you're even entirely ignorant of basic concepts like what a latest tag is supposed to represent and how tags are used to update apps.
The fact that you try to pass off that absurd assertion in such absolute terms makes it quite clear that you have absolutely no idea about what you're talking about.
Hell, your claim can only possibly be made by someone who never even registered a user account in Docker Hub. Please refrain from speaking about stuff you know nothing about.
> If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling
This mindless statement just makes it clear as water that you have no idea of how containers are used. At all. And yet here you are degrading the signal/noise ratio by blabbering about stuff you know nothing about.
I don't know what else to say. I work with docker containers almost daily. We keep our own images in a private registry (GitLab, since we already pay for it) and those images are built on top of public images from the Hub (postgres, python, etc...). Our CI server pulls the base images from the Hub (keeps a cache), the source from GitLab, does the build and pushes the resulting images back to the private registry. Another server gets called when this happens to spin up a testing environment (pulling our images from the private and some 3rd party images from the public repo - again, cached). Automated and human testing is done against that and if it all passes, the signal is sent to the production server to update itself (doing on final round of pulls from both repos - often only the private one, as our app images are updated far more often than supporting services like postgres). Unless the base or supporting images are updated, the Hub receives nothing but a few HEAD requests from us and a push of that one FOSS image that we actually release publically.
Either way, I can't be bothered with this thread anymore as there's clearly no more constructive discussion to be had with you. Cheers
> If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong.
Most CI systems poll to see if there are new versions of an image. If this is done with a GET request (to get the manifest as a first step towards pulling layers), it counts towards the rate limiting.
> Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.
Maven separates the domain of a Maven repository from the coordinates of packages. Docker does not. Dockerhub's domain is inferred into any shorthand reference like `foo/bar`, meaning that it will ignore a pull-through cache.
You can ostensibly configure a docker daemon to treat another registry as a trusted mirror, but it's harder than it looks in practice, because on most production systems that won't (and shouldn't) be enabled.
Ok, if just checking whether a newer version is available counts towards the rate limit, than that is just completely insane!
EDIT: Yeah, that makes sense given the protocol's design. Anyways, HEAD requests don't count, so hopefully the client can utilise those (or be modified to). [/EDIT]
And yes, Maven has a design much better suited to having a local "proxy" (in fact, it always does that on localhost anyways), but the setup looks pretty straightforward and I really don't see much reason to not use it on all dev, test and CI machines. I'd also use it in production, but even if you'd prefer not to, production servers don't pull that often and updates are usually done through hooks, not constantly polling a repo.
So basically, from my perspective (handful of servers, each with some 20 containers) this change will be thoroughly annoying, but will force me to set up caching which will make things much faster and also decrease the reliance on an external service for day-to-day operation. Not a fan of it, but yi completely see the reason why they did it.
My point is that the name of the image and the location of the registry are mixed together. If it was easy to do cache hierarchies folks wouldn't be so fussed.
The biggest issue is that if you run the command 'docker pull X', it counts as a pull even if you already have the up-to-date image, and so it doesn't do a download.
25 comments
[ 0.23 ms ] story [ 73.0 ms ] threadhttps://www.youtube.com/watch?v=WpE_xMRiCLE
I have to admit that 100 pulls per 6 hours doesn't sound like enough for even a small team doing CI/CD.
But in the long run, it might make a lot of organizations more lean in how they do builds, which could be a good thing.
In my POV the first fundamental change that this caused was to cease considering docker hub as reliable infrastructure, and therefore started to look for solutions to reliably source container images. For instance, GitLab already provides its own container registry, and thus it makes sense to simply source images from them than from tertiary sources like Docker Hub.
Another impact that this has is that I'm looking into the need to operate a custom container registry for each Kubernetes cluster. It wouldn't be nice if an application running on Kubernetes or Docket Swarm risked downtime just because you happened to hit one of Docker Hub's rate limits.
I do recall reading about the free pulls a year ago and thinking, "Wow, that's awfully generous!" :)
We don't need to resort to paid services. This is all about convenience.
As I've said, services like GitLab already provide container registry services, and we are also free to operate our own custom container registry services. Docker Hub is only used to source reference images such as Ubuntu, postgres, rabbitmq, traefik, etc etc etc. We can simply point to a different provider to continue accessing those. Even if these projects continue to use Docker Hub then if we stick with stable releases then we can simply point to a different container registry service, even one managed by us.
The main point is that Docker Hub ceased to be a reliable provider. Thus, we plan accordingly.
Maybe we understand something different by a "small" team, but IMHO a smaller team uses more ready public images, to save time and costs. Also even not that big project on Kubernetes that includes internal software will also have a lot of other software.
Let's speculate: I had a look at some teams I know and how they work (or they would like to work soon) and it's easily 15-30 images in a CI/CD pipeline. Plus tools used to develop, but not exactly released in a build (source code management, PM, communication etc). All teams I know have 2-3 cycles developed at all times (next release, hotfix, future release after the next one). This means a lot of these tools are multiplied by 2-3 (upgrading tools as well, so multiple versions of same tool needed). If you have a few clusters that have different requirements it can also be multiplied by 2 or 3.
I would not dare to think what would happen if someone used "latest" tag in their daily work, or organized a docker/kubernetes workshop for new people :)
It really depends on your usecase. I mean, each CICD stage can involve building a brand new image and running a couple of them, sometimes more. 100 pulls per 6 hours corresponds to a constant rate of 3 min between pulls, which would require between 15 to 20 minutes for each commit.
Anyway, the primary practical implications of this sort of change is a) rate limiting suddenly becomes a requirement, b) we can no longer count on Docker Hub to provide a reliable service, thus we plan accordingly.
I may be wrong but I interpreted that to mean that even caching won't prevent you from hitting the limit.
Just a few years back I remember even the smallest Java shop would have a local "caching/forwarding" Maven repository and the upstream repos were only considered a fallback.
Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.
[0] https://docs.docker.com/registry/recipes/mirror/
A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc.
I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.
Yes.
> If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull.
Yes.
> Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?).
Not necessarily. You dockerize an app prior to running non-unit tests tests, and that image needs to be stored by a container registration service.
With this change, Docker Hub's ceases to be that service.
> You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
Nope, the point is that Docker Hub ceases to be relevant as a container registry. As long as you start to do all your work outside of Docker Hub, you won't need Docker Hub to do any of the work. You just get your base images from where you store your own container images and suddenly Docker Hub stops playing any role in your workflow, thus losing al relevance.
If you're in a situation where having your own registry means Docker Hub "loses all relevance", then you're probably not the kind of user they have any reason to care about (and that's nonsense anyway, because you'll still almost certainly be using 3rd party images from there).
I'm sorry to say but your insistence of speaking out of a position of blisful ignorance and your refusal to open your eyes does not change the world around you.
> but I've always seen Docker Hub as...well...a hub, not just any ol' registry.
No one with the faintest cursory knowledge of Docker would ever claim that Docker Hub is anything other than a docker registry.
Docker itself advertises Docker Hub as a service "to create, manage, and deliver your teams' container applications." Docker itself describes Docker Hub as a "cloud-based application registry and development team collaboration services." That's their own official marketing slogan, taken straight out of their "Why Docker" page.
And even that you managed to miss and failed to understand.
> It's where people publish final builds
It really isn't. That's not how containers work at all, and that's not how Docker promotes the service, and that's not how anyone uses it. Nothing in your comments is even remotely compatible with basic notions of CICD.
Apparently you're so out of touch with containerization and CICD and devops workflow that you're even entirely ignorant of basic concepts like what a latest tag is supposed to represent and how tags are used to update apps.
The fact that you try to pass off that absurd assertion in such absolute terms makes it quite clear that you have absolutely no idea about what you're talking about.
Hell, your claim can only possibly be made by someone who never even registered a user account in Docker Hub. Please refrain from speaking about stuff you know nothing about.
> If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling
This mindless statement just makes it clear as water that you have no idea of how containers are used. At all. And yet here you are degrading the signal/noise ratio by blabbering about stuff you know nothing about.
Either way, I can't be bothered with this thread anymore as there's clearly no more constructive discussion to be had with you. Cheers
Most CI systems poll to see if there are new versions of an image. If this is done with a GET request (to get the manifest as a first step towards pulling layers), it counts towards the rate limiting.
> Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit.
Maven separates the domain of a Maven repository from the coordinates of packages. Docker does not. Dockerhub's domain is inferred into any shorthand reference like `foo/bar`, meaning that it will ignore a pull-through cache.
You can ostensibly configure a docker daemon to treat another registry as a trusted mirror, but it's harder than it looks in practice, because on most production systems that won't (and shouldn't) be enabled.
EDIT: Yeah, that makes sense given the protocol's design. Anyways, HEAD requests don't count, so hopefully the client can utilise those (or be modified to). [/EDIT]
And yes, Maven has a design much better suited to having a local "proxy" (in fact, it always does that on localhost anyways), but the setup looks pretty straightforward and I really don't see much reason to not use it on all dev, test and CI machines. I'd also use it in production, but even if you'd prefer not to, production servers don't pull that often and updates are usually done through hooks, not constantly polling a repo.
So basically, from my perspective (handful of servers, each with some 20 containers) this change will be thoroughly annoying, but will force me to set up caching which will make things much faster and also decrease the reliance on an external service for day-to-day operation. Not a fan of it, but yi completely see the reason why they did it.