Docker Hub rate limits behind a virtual Docker repository
Acme's platform team runs Artifactory 7.71 on-prem. Docker repositories are docker-hub-remote pointing at https://registry-1.docker.io, dev-docker-local, prod-docker-local, and docker-virtual, which lists dev, prod, then remote. The virtual repo's default deployment repo is dev-docker-local. About 400 CI jobs per day pull alpine:3.19, node:20.11-alpine, and python:3.12-slim through docker-virtual; production Kubernetes pulls the same virtual repo. Some base layers are definitely cached because repeated pulls are fast, but several jobs per hour fail with toomanyrequests: You have reached your pull rate limit. The remote repo is configured anonymously. A recent release promoted payments-api:2.4.0 from dev to prod, but a production pod still pulled 2.4.0 with the previous digest for 20 minutes. The team wants to reduce Docker Hub calls and make promotion predictable without giving CI admin rights.
1. Why can Docker Hub rate-limit errors appear even when layers are cached in `docker-hub-remote`? What configuration and client changes would you make first?
Read the worked answer
What is happening
Artifactory caches Docker layers by digest, but a pull of node:20.11-alpine first needs the tag-to-digest manifest. If that manifest is not cached, or its metadata cache period has expired, the remote repo calls Docker Hub again. Layers can be local while manifest lookups still consume the anonymous rate limit. Also confirm clients are not pulling directly from registry-1.docker.io; direct pulls bypass the cache entirely.
First changes
- Put a Docker Hub service account on
docker-hub-remote(and enable token authentication if required) so requests are authenticated and get a higher limit. - Raise the remote repo metadata retrieval cache period to a sensible value instead of zero; do not set the artifact cache to purge aggressively.
- Pin base images by digest in CI and Kubernetes, and use
imagePullPolicy: IfNotPresentfor immutable digests. Tag pulls may still revalidate. - Check the remote repo advanced settings and request logs for 429 responses to distinguish Artifactory misses from direct client pulls.
Do not treat a cached layer as proof that the tag lookup is cached; the manifest path is the rate-limited part.
2. The virtual repo order is dev, prod, remote. Explain the resolution risk for production and propose a repository layout that keeps CI and runtime isolated while still using one remote cache.
Read the worked answer
Risk
Virtual repository resolution is ordered. With dev-docker-local first, a production pull of payments-api:2.4.0 resolves to the dev copy if that path and tag exists, even after promotion. The default deployment repo only controls where docker push goes; it does not affect read order. One virtual repo cannot give CI and production different precedence.
Layout
Create two virtual repos:
docker-ci-virtual:dev-docker-localfirst (default deployment), thendocker-hub-remote. CI pushes and pulls app images here and gets base images from the remote.docker-prod-virtual:prod-docker-localfirst, thendocker-hub-remote. Production is read-only and cannot see dev tags.
Give the CI group deploy and read on dev-docker-local, and read on docker-ci-virtual; give the runtime identity read-only on docker-prod-virtual. Promotion copies the image to prod-docker-local through a controlled job, not by making prod virtual include dev. If a single virtual repo is mandatory, put prod-docker-local before dev-docker-local, but accept that CI may resolve stale production tags; separate virtual repos are safer.
3. After promotion, why might a pod still run the old digest for `payments-api:2.4.0`, and how should the promotion process and tag strategy change?
Read the worked answer
Why the old digest persists
Docker tags are mutable. Promotion can copy the same tag to prod-docker-local, but a node that already has payments-api:2.4.0 may not re-pull if the tag is unchanged and imagePullPolicy is IfNotPresent. If the pod does re-pull, the old virtual order (dev before prod) can still resolve the dev tag. Artifactory cannot invalidate a kubelet's local image store.
Process change
- Tag every build immutably, e.g.
2.4.0-b20241118-<git-sha>, and never overwrite a released tag. - Promote that unique tag by copy to
prod-docker-local; record the manifest digest returned by the promotion. - Deploy by digest in the production manifest, or update the deployment to the new unique tag. This removes ambiguity even if a tag is later reused.
- Keep production on
docker-prod-virtualonly. Verify by pulling the digest from a clean node and comparing it with the promotion output. - Add a CI check that the digest in the deployment matches the digest in
prod-docker-local; fail the release if not.