When you run a Kubernetes cluster in production and you happen to misconfigure a deployment it can quickly happen, that you hit the Docker Hub rate limits. But it can also happen, that an upstream registry is just unavailable or an image has been deleted. In order to counter this, you can set up harbor, a common Kubernetes registry, which allows to transparently cache content. With cri-o configured to automatically check this registry, you can get automatic mirroring done, without touching any manifest or manually editing image strings.

Before you start

It’s expected that you already have harbor deployed in your cluster. It’s easy to deploy, so I’ll leave it to you. Obviously it’s also expected that your Kubernetes cluster is already running and that it’s using cri-o as container runtime.

Set up a cache project

Setup a registry endpoint for e.g. Docker Hub, without authentication.

Next setup a project that is configured as “Proxy Cache” for the endpoint created in the previous step. In this case we call it dockerhub. If you want, you can setup a resource quota, which helps to keep the storage use limited.

Configure cri-o to use the cache

Now it’s time to configure cri-o to resolve the image registries automatically without being reconfigured. This is done by using the registries.conf’s mirror feature.

Just take the following snippet, replace <your harbor registry> with the domain of your harbor setup, and drop it at /etc/containers/registries.conf.d/mirrors.conf on each of your Kubernetes nodes:

[[registry]]
prefix = "docker.io"
location = "docker.io"
  [[registry.mirror]]
  location = "<your harbor registry>/dockerhub"

Now restart cri-o (or the node altogether) and it should be picked up automatically.

Note: This will only take care of images that are pulled without credentials.

Next Steps

With the basics in place and basically no downsides, since cri-o will automatically try the local mirror, and if that fails pull directly from upstream, you can explore all other repositories in your cluster and add mirrors for these as well.

Use this snippet from the Kubernetes manual which will give you all images currently running in your cluster and check the registries used:

kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c

Alternatively, you can ask your local Prometheus instance for the images used in the cluster using following query:

count(label_replace(kube_pod_container_info, "image", "$1", "image", "([^/]+)/.*")) by (image)

Just repeat the steps above with the other registry and quickly you’ll have a full mirror of all images in your cluster.

Conclusion

It’s very easy to get a full mirror of images locally in your cluster, when utilising the cri-o features. (These work for podman as well, by the way.) Which makes your cluster a lot more resilient regarding outside influences. You also get vulnerability scanning for free.