Mutable Image Tags Break the Rollout and the Rollback

Two independent things are going on, and both must be checked.

1. The Deployment template may not have changed at all. kubectl set image rewrites spec.template.spec.containers[*].image. If the resulting string is byte-identical to what is already there (...:v1.4), the pod template hash does not change, no new ReplicaSet is created, and no rollout happens. rollout status then immediately reports success because there is nothing in flight - it is reporting on the old, already-complete rollout. That alone explains prod.
2. Even if the template had changed, the tag is a cache key. With no explicit policy, a non-:latest tag defaults to IfNotPresent. Prod's nodes already have v1.4 cached from three days ago, so the kubelet reuses it. Staging's nodes were replaced, the cache was empty, and the pull fetched whatever v1.4 points to now - the hotfix.

The successfully rolled out message is therefore not evidence that new code is running. It is evidence that the Deployment's desired state matches its observed state, and the desired state still names the same mutable string.

It is effectively broken for this pipeline, for the same reason the forward deploy was.

rollout undo restores the previous ReplicaSet's pod template. If every release in history used :v1.4, then the previous revision's image string is also :v1.4. Rolling back produces a template that differs from current in nothing that affects the image, and on any node with the layer cached you get the same bytes back. The rollback appears to succeed and changes nothing.

Consequences to state plainly:

• Rollback capability is a property of the artifact scheme, not of the kubectl verb. Immutable references are the precondition.
• revisionHistoryLimit (default 10) bounds how far back you can go; with mutable tags the history is meaningless anyway.
• Tag drift is undetectable after the fact. Nobody can answer "what was in prod at 14:00?" because the tag no longer identifies a unique image.

Interim mitigation while the pipeline is fixed: pin prod to an immutable reference manually (kubectl set image ...=registry.example.com/payments@sha256:<digest>) so at least the current state is knowable and reversible.

Target model: the tag is an address, the digest is the identity.

• Build once, tag with the git SHA or a monotonically increasing build number (payments:git-a1b2c3d), and record the digest in the pipeline output.
• Promote the same artifact through dev → staging → prod; never rebuild per environment, or you reintroduce the staging/prod divergence the team just hit.
• Deploy by digest in prod (payments@sha256:...) or by immutable tag with an admission policy that rejects mutable tags. This is enforceable with a Kyverno/OPA rule rather than a convention.
• Prefer a GitOps controller reconciling a manifest repo, so the cluster state is auditable and drift is detected - with kubectl set image from CI there is no record of what was applied when.

On :latest + Always: it does force a registry check on every pod start, so nodes will not serve stale bytes. But it fails the requirements that actually matter here:

• Rollback still cannot express "the previous image", because the previous revision also says :latest.
• It couples pod startup to registry availability - a registry outage becomes a restart outage - and adds pull latency to every scale-up.
• It makes the running version unknowable from the manifest; you must inspect the live pod spec or imageID.

Immutable references give cheap pulls from cache, unambiguous rollback, and an auditable answer to "what is in prod?" - none of which Always provides.