DaemonSet Rollout Gaps and a Security-Driven Agent Upgrade Under Freeze
Northwind Retail runs Instana SaaS for a self-managed Kubernetes estate: 3 regions, 240 worker nodes, and about 12,000 pods. The Instana agent is deployed as a Helm-managed DaemonSet and reconciled by Argo CD; direct kubectl edits are prohibited. An audit finds 58 nodes in the payments node pool have no Instana host entry. That pool uses the taint dedicated=payments:NoSchedule. The DaemonSet currently reports 182 desired/ready pods. Separately, 120 nodes still run agent 1.2.5 while 62 run 1.2.8; security requires 1.2.8 before a change freeze begins in 36 hours. Several agent pods are OOMKilled with a 1Gi memory limit. The team cannot restart application pods and must keep the rollout GitOps-compatible. The platform team owns the Helm chart; application teams own namespaces.
1. Why do the 58 payments nodes lack Instana host entries, and what GitOps-compatible change would you make? Give the reasoning and a verification step.
Read the worked answer
The likely cause is scheduling, not an agent crash: a DaemonSet only places pods on nodes that satisfy its tolerations and node affinity. The payments pool has dedicated=payments:NoSchedule, and the agent DaemonSet presumably has no matching toleration. The arithmetic supports this: 240 - 58 = 182, exactly the desired/ready count. I would confirm with kubectl describe daemonset <agent> and kubectl get pods -n instana-agent -o wide that there are zero agent pods on those nodes rather than CrashLoopBackOff.
Because Argo CD owns the release, I would add the toleration in the Helm values-key dedicated, operator Equal, value payments, effect NoSchedule-and let Argo sync. I would not kubectl patch the DaemonSet. I would also check whether the OOMKilled pods sit on the same pool: if the 1Gi limit is too low, fix memory in the same values change so newly scheduled pods do not crash.
Verification: Argo sync succeeds, desired/ready reaches 240, and Instana host count for the payments zone rises by 58 within a few minutes.
2. Sequence the 1.2.5→1.2.8 upgrade in 36 hours. What do you do first, what can be canaried, and what do you do if the freeze arrives with nodes remaining?
Read the worked answer
First fix the memory limit and the toleration, because both affect upgrade safety; an OOMKilled agent during the rollout looks like a version failure and wastes the window. Then change the image tag in Helm values to 1.2.8 and let Argo reconcile. Use a conservative rolling update (for example, maxUnavailable of 10% or a small fixed number) so the DaemonSet replaces pods node by node; do not delete pods manually.
Canary by node pool or region: start with one non-payments pool, verify agent health and host metrics, then move to payments, then the remaining regions. Application pods are not restarted, so the risk is monitoring gaps rather than workload downtime, but production waves should still avoid peak. If the freeze arrives with nodes remaining, do not claim completion. Document the exception, prioritize the security-relevant or highest-risk nodes, and open an emergency change for the rest. A CVE may justify a freeze exception, but that is an owner risk decision. Roll back by reverting the image tag in Git if the new agent crashloops or host metrics drop sharply.
3. What checks prove the rollout is healthy, and what would make you roll back?
Read the worked answer
Healthy: DaemonSet desired=ready=240; every node has an agent pod; Instana shows agent version 1.2.8 on all 240 hosts; payments hosts now report host and container metrics; no OOMKilled or CrashLoopBackOff events; agent status is connected; and missing-host alerts do not spike. I would compare host counts and metric gaps before/after and spot-check a payments node in the UI.
Rollback triggers: agent pods crashloop beyond the canary, host count falls instead of rises, metrics stop for a whole pool, or OOMKills continue after the memory increase. Rollback should be a Git revert of the image tag, not a manual edit, so Argo restores the previous desired state.