It bugs me that this implementation detail of containerd has leaked to such an extent. This should be part of the containerd distribution, and should not be pulled at runtime.
Instead of just swapping out the registry, try baking it into your machine image.
Relying on an hosted image also caused some disruptions for Nomad (the scheduler from Hashicorp), because the default pause image was hosted at gcr.io which google killed, and it moved to registry.k8s.io.
I've used k8s before a lot and at several companies. I am convinced that 99.9% of the people who use it should not be. But it's more fun than deploying VM images at least.
Same here, I went through a few projects since 2021 where doing Kubernetes setups were part of my role on the consulting project, and I would say prefer managed containers solutions, e.g. Azure Web Apps, or when running locally plain systemd or Docker Compose.
Anything else, most companies aren't Web scale enough to set their full Kubernetes clusters with failover regions from scratch.
O/T, but I'm getting a cert error on this page - wonder if it's just me or if this site is just serving a weird cert. Looks like it's signed by some Fortinet appliance - maybe I'm getting MITMed? Would be kind of exciting/frightening if so.
EDIT: I loaded the page from a cloud box, and wow, I'm getting MITMed! Seems to only be for this site, wonder if it's some kind of sensitivity to the .family TLD.
Nice to know, though I wonder how many companies are really using all private images? I've certainly had a client running their own Harbor instance, but almost all others pulled from Docker Hub or Github (ghcr.io).
Lots of medical and governmental organisations are not allowed to run in public cloud environments. It's part of my job to help them get set up. However, in reality that often boils down to devs wining about adding a registry to Harbor to cache; nobody is going to recompile base images and read through millions of lines of third party code.
A lot of security is posturing and posing to legally cover your ass by following an almost arbitrary set of regulations. In practice, most end up running the same code as the rest of us anyway. People need to get stuff done.
The Public Sector and anyone concerned with compliance under the Cyber Resilience Act should really use their own private image store. Some do, some don't.
Just to save someone 5 minutes of research, if you are using the EKS AMIs based on AL2023 or Bottlerocket, this is already done for you by pointing to an image on ECR. At least on Bottlerocket, I haven't checked AL2023, the image is baked into the AMI so you don't even need to pull it from ECR.
I believe this has been patched time and time again in on-premises variants like OpenShift. Curious to check if it’s there in small variants like microk8s, k3s, etc., as I’m considering moving a few offline services to Talos.
Yeah that pause image was really annoying when I was hosting a k8s cluster on Hetzner, since the `registry.k8s.io` -registry was blocking some Hetzner IPs, since its hosted on Google.
Also, if you are using kubeadm to create your cluster, beware that kubeadm may be pre-pulling a different pause image if it does not match your containerd configuration:
https://github.com/kubernetes/kubeadm/issues/2020
This is insane, am I the only one being shocked to learn about this? This reeks of bad engineering, is there at least a plan to make this go away by embedding "pause" with the distribution?
21 comments
[ 2.1 ms ] story [ 42.0 ms ] threadInstead of just swapping out the registry, try baking it into your machine image.
The nomad team made this configurable afterwards.
Anything else, most companies aren't Web scale enough to set their full Kubernetes clusters with failover regions from scratch.
EDIT: I loaded the page from a cloud box, and wow, I'm getting MITMed! Seems to only be for this site, wonder if it's some kind of sensitivity to the .family TLD.
A lot of security is posturing and posing to legally cover your ass by following an almost arbitrary set of regulations. In practice, most end up running the same code as the rest of us anyway. People need to get stuff done.
https://github.com/awslabs/amazon-eks-ami/pull/2000
There was a discussion open on containerd's GitHub on removing the dependency on the pause image but it has been closed as won't fix: https://github.com/containerd/containerd/issues/10505
Also, if you are using kubeadm to create your cluster, beware that kubeadm may be pre-pulling a different pause image if it does not match your containerd configuration: https://github.com/kubernetes/kubeadm/issues/2020