ECR is kind of hard to beat if you're ok with being in the cloud.
The last time I used it earlier this year for a company already on AWS, it was ~$3 / month per region to store 8 private repos and it was really painless to have a flexible and automated life cycle policy that deleted old image tags. It supports cross region replication too. All of that comes without maintenance or compute costs and if you're already familiar with Terraform, etc. you can automate all of that in a few hours of dev time.
For what it's worth, I have an on-prem Nexus server with a Docker repository. It has 8 cores and 16GB RAM. It has 82000 hits/day in the webserver log, though 99.9% of them transfer only a few kB, so I assume it's a metadata check and the client already has the correct version.
The same Nexus is also hosting our Maven and NodeJS repositories. That has 1,800,000 hits per day, although all but 120,000 of them return HTTP 404. (I think one of our clients has misconfigured their build server to use our Maven repository as a mirror of Maven Central, but as it's just an AWS IP I don't know who it is.)
I'm sure it's overprovisioned, but the marginal cost of 2, 4 or 8 cores and 4, 8 or 16GB RAM isn't much when we buy our own hardware.
I also run Harbor. I use the official Helm chart; it's a little jank, doesn't support a couple of things we want. It only works with one of ArgoCD/ExternalSecretsOperator, and it doesn't support Redis TLS.
Contrary to the author of this post, we just run one (the "source of truth") and use caching proxies in other regions. Works fine for us.
This looks nice. What would be good on-prem S3 companion for this? I know if minio but I think there was some recent drama about it (I don't know specifics, just a feeling)
We are using deepspace storage for this. We can get/put objects into the cloud as one of our target volumes. It works as an auto archive writing anything in the fs that is over 90 to a compressed object and leaving behind a stub. You can point Harbor (or other tools like Mimir/Loki ) to a DeepSpace endpoint which looks like a standard S3 target. Then add policies and the files get moved, replicated, versioned to tape, cloud, disk array in the background. The users just interact with the file system as usual and admins have a UI with a catalog which shows where everything actually is.
I'm confused on why they decided to populate the cache by replicating the entirety of Docker Hub instead of using a sort of cache that gets populated on the first pull
We self-host Harbor as well, it’s fairly painless. Has SSO out of the box, a Terraform provider that covers everything, and for the most part just works.
The issues we’ve had so far:
- No programmatic way to retrieve your token that’s required for ‘docker login’. So we had to create a robot account per user and pop their creds into our secrets store.
- Migrating between sites by cloning the underlying S3 bucket and spinning up the new Harbor instance on top of it, does not work. Weird issues with dropping pulls.
- RBAC goes down to project, not repository level, complicating some of our SDLC controls.
- CSRF errors every time you try to do anything in the UI
- Lenient API and lack of docs means things like setting up tag immutability rules via Terraform was a bit of a PITA to figure out the right syntax
So some small issues, but definitely a great piece of software.
> pulling and pushing our images over the internet dozens of times a day caused us to hit the contracted bandwidth limit with our datacenter provider Deft repeatedly
I wonder what they were doing that resulted in blowing out their Docker layer cache on every pull and push.
Normally only a layer diff would be sent over the wire, such as a code change that didn't change your dependencies.
So what are the thoughts of folks who have used Nexus and moved to Harbor?
In my experience Nexus is a bit weird to administer and sometimes the Docker cleanup policies straight up don't work (junk left over in the blob stores even if you try to clean everything), but it also supports all sorts of other formats, such as repositories for Maven and NuGet. Kind of hungry in regards to resources, though.
I am not to familiar with Kamal but it seems possible to integrate it with my project Spegel to remove some of the load from upstream. Especially if they are running clusters of servers physically located close to each other they could avoid some of the replication complexity with multiple Harbor instances.
Naive question: why not put the effort into building the RoR apps into a binary and run the all services with systemd? No need to deal with Docker and the entire ecosystem around it
23 comments
[ 4.9 ms ] story [ 30.5 ms ] threadThe last time I used it earlier this year for a company already on AWS, it was ~$3 / month per region to store 8 private repos and it was really painless to have a flexible and automated life cycle policy that deleted old image tags. It supports cross region replication too. All of that comes without maintenance or compute costs and if you're already familiar with Terraform, etc. you can automate all of that in a few hours of dev time.
They never say that it needs these resources, they say that the current VM has this config. Probably overkill by 1000%.
The same Nexus is also hosting our Maven and NodeJS repositories. That has 1,800,000 hits per day, although all but 120,000 of them return HTTP 404. (I think one of our clients has misconfigured their build server to use our Maven repository as a mirror of Maven Central, but as it's just an AWS IP I don't know who it is.)
I'm sure it's overprovisioned, but the marginal cost of 2, 4 or 8 cores and 4, 8 or 16GB RAM isn't much when we buy our own hardware.
Contrary to the author of this post, we just run one (the "source of truth") and use caching proxies in other regions. Works fine for us.
The issues we’ve had so far:
- No programmatic way to retrieve your token that’s required for ‘docker login’. So we had to create a robot account per user and pop their creds into our secrets store.
- Migrating between sites by cloning the underlying S3 bucket and spinning up the new Harbor instance on top of it, does not work. Weird issues with dropping pulls.
- RBAC goes down to project, not repository level, complicating some of our SDLC controls.
- CSRF errors every time you try to do anything in the UI
- Lenient API and lack of docs means things like setting up tag immutability rules via Terraform was a bit of a PITA to figure out the right syntax
So some small issues, but definitely a great piece of software.
One glaring omission is lack of support for proxy docker.io without the project name i.e pulling nginx:latest instead of /myproject/nginx/nginx:latest
The workaround involves URL rewrite magic in your proxy of choice
I wonder what they were doing that resulted in blowing out their Docker layer cache on every pull and push.
Normally only a layer diff would be sent over the wire, such as a code change that didn't change your dependencies.
In my experience Nexus is a bit weird to administer and sometimes the Docker cleanup policies straight up don't work (junk left over in the blob stores even if you try to clean everything), but it also supports all sorts of other formats, such as repositories for Maven and NuGet. Kind of hungry in regards to resources, though.
One issue to solve is auto-deletion of old images, so that the storage does not swell. Any tips?
It is pretty easy to just run the basic registry for this purpose.
We have a similar setup for NPM and Pypi on the same machine. It doesnt really need a lot of attention. Some upgrades every once and a while.
This is incorrect, the word you are looking for here is "on-premises" - a "premise" is something entirely different.