CGOA: Argo CD with selfHeal Disabled: Drift, Sync Status, and Reconciliation Vocabulary

Argo CD with selfHeal Disabled: Drift, Sync Status, and Reconciliation Vocabulary

Northwind Retail runs 14 Kubernetes clusters (3 production, 4 staging, 7 edge) with Argo CD. Applications sync automatically, but the ApplicationSet sets syncPolicy.automated.selfHeal: false because, per the platform lead, "we don't want the controller fighting on-call during incidents." Last month an engineer patched a Deployment image with kubectl set image during a payment outage; the patch stayed live for nine days and no commit was ever made to match it. In the postmortem the words "drift", "out-of-sync" and "reconciliation" were used interchangeably. A second argument concerned a cluster whose Application shows Synced but Degraded after a node ran out of memory; someone claimed "GitOps failed, it did not fix it." A third question was whether the nine-day divergence would have been visible at all in the Argo CD UI, and whether turning selfHeal back on would have prevented the incident or made it worse.

1. Classify each item using precise GitOps terminology and justify: (a) the `kubectl set image` patch, (b) Argo CD's periodic comparison of live state to Git, (c) the Synced-but-Degraded Application.

Read the worked answer

(a) The patch is drift - a divergence between desired and live state. Desired state is the manifest set in Git (image tag A). Live/actual state is what the API server holds (tag B). The kubectl write changed live state without changing desired state, which is exactly drift. Argo CD surfaces it as OutOfSync, but "out-of-sync" is the reported symptom; drift is the condition. Note the drift is of two kinds: the live object no longer matches Git, and the running system is no longer described by any committed artifact - meaning Git stopped being a faithful record of production.

(b) The comparison is the observation half of reconciliation. Reconciliation = observe actual state, compare to desired state, act to converge. Argo CD compares cached rendered manifests from the repo-server against live manifests on a loop (default 180s, plus webhooks). Comparison alone is detection, not convergence; the corrective step is the sync, which with selfHeal: false only happens on a Git change or a human pressing Sync. So reconciliation here is detection-only, deferred to humans.

(c) Synced + Degraded is not a GitOps failure - the statuses are orthogonal axes. Sync status answers "does live match Git?" Health answers "is the workload functioning?" A Degraded-but-Synced app means the desired state was applied faithfully and the runtime failed (OOM, bad probe, insufficient node capacity). Converging live to desired state cannot fix a desired state that is itself unhealthy or a cluster that cannot schedule it. The correct follow-up is a Git change (raise limits, fix the manifest) or a cluster capacity fix, not a controller change.

2. Would the nine-day divergence have been visible in the Argo CD UI? Explain the mechanism, and assess whether `selfHeal: false` is compatible with GitOps and what controls you would use instead.

Read the worked answer

Visibility: yes, almost certainly. The application controller re-compares on its reconciliation interval, so within minutes of the kubectl set image the Deployment would render as OutOfSync with the diff shown in the UI, and the resource would stay flagged for all nine days. The divergence persists, not because it is invisible, but because nothing acts on it. Two caveats worth checking in a real incident: the resource must be tracked by an Application (not orphaned or excluded by ignoreDifferences), and a wildcard/RespectIgnoreDifferences setting or a resource excluded from the app's source can mask it.

Compatibility with GitOps. The fourth principle asks that software agents continuously observe actual state and attempt to reconcile it with desired state. Detection without any convergence attempt weakens that to "Git is the source of truth for what we intend, until a human syncs." The nine-day window is the real violation: production ran in a state no commit described, and nothing escalated.

Better controls than a blanket selfHeal: false:

Keep selfHeal on and use sync windows to suppress automated sync during defined incident/freeze periods, so the controller does not fight on-call at the worst moment but resumes afterwards.
Use ignoreDifferences narrowly (e.g., HPA-managed replica counts, mutating-webhook fields) instead of disabling repair globally.
Alert on OutOfSync older than N minutes, and require a commit within the incident timeline (break-glass: apply, then commit the same change immediately).
Remove or tightly scope human write access to production so drift has fewer sources. The judgement call: selfHeal off is a legitimate availability tradeoff, but only paired with alerting and a commit obligation - otherwise drift becomes permanent and Git becomes documentation rather than source of truth.