Registry Retention Policy Deletes a Needed Layer
An internal registry at registry.internal:5000 stores images for 40 services. Disk is at 91%. An admin enables a garbage-collection and retention job that keeps the last 10 tags per repository and deletes untagged manifests older than 7 days, running nightly at 02:00. Two days later, a production rollback to api:1.9.3 fails with manifest unknown, even though api:1.9.3 was tagged and pushed three weeks ago. A postmortem shows the tag was overwritten by a re-push of api:1.9.3 during a hotfix, leaving the original manifest untagged. The registry runs without a read-only window during GC. Users authenticate with a shared robot account that has push and delete rights.
1. Explain how a tagged image became unreachable despite the retention policy nominally keeping the last 10 tags.
Read the worked answer
The retention rule keeps tags, but a tag is a pointer to a manifest. When api:1.9.3 was re-pushed during the hotfix, the tag was moved to a new manifest. The original manifest lost its only tag and became untagged. The policy deletes untagged manifests older than 7 days, so on the next nightly run the original manifest - and its unique layers - were removed.
Later, the rollback referenced api:1.9.3, but by then the tag pointed at the hotfix manifest, or the rollback tooling resolved a digest that no longer existed. The error manifest unknown is consistent with a digest whose manifest was collected. The lesson: retention based on tags does not preserve history. Immutable references (digests) and tag immutability settings are what prevent this class of loss.
Also note the GC ran without a read-only window. Concurrent pushes during GC can race, and a manifest pushed mid-run may be partially collected or left inconsistent.
2. Recommend changes to retention, access, and GC procedure that prevent recurrence while still controlling disk growth.
Read the worked answer
Retention and immutability
- Enable tag immutability for release tags so
1.9.3cannot be silently repointed; require a new tag for a hotfix. - Pin deployments by digest, not mutable tag, so rollback targets a manifest that retention will not collect.
- Keep untagged manifests longer (e.g., 30 days) and exempt manifests referenced by any active deployment record.
Access
- Replace the shared robot account with scoped accounts: CI pushes, deploy tooling pulls, and only a break-glass account deletes. A shared push+delete account makes accidental or malicious deletion untraceable.
- Enable audit logging on delete operations.
GC procedure
- Run GC in a read-only window (pause pushes) or use a registry that supports online GC safely.
- Run a dry-run first and record what would be deleted.
- Alert on disk thresholds earlier (e.g., 75%) so retention is not tuned under emergency pressure.
Together these convert a destructive, irreversible policy into one that reclaims space while preserving rollback capability.