Containerd Migration Causes Missing Container Metrics on a Subset of Nodes
Fabrikam Logistics monitors 60 Kubernetes nodes and about 1,200 containers with Instana. Host metrics are complete, but 180 containers on 12 nodes do not appear in container dashboards. Those 12 nodes were migrated from Docker to containerd 1.7 last month. The Instana agent runs as a DaemonSet. Security policy forbids privileged pods and writable hostPath mounts; read-only hostPath mounts are allowed with review. The current DaemonSet mounts /var/run/docker.sock read-only. Application teams report that container CPU and memory are missing, which breaks capacity dashboards. You need to restore visibility without weakening the security posture or creating duplicate container entities. The cluster uses Kubernetes 1.27 and the agent version is 1.2.8 on all nodes. No application pods can be restarted for this change.
1. What is the most likely cause of the missing container metrics, and what evidence would you collect before changing the DaemonSet?
Read the worked answer
The 12 migrated nodes now use containerd, but the agent DaemonSet only mounts the Docker socket. Host metrics come from cgroups/kernel interfaces and can still work, while container metadata and per-container CPU/memory need the container runtime API. The pattern-180 missing containers on exactly those 12 nodes-is the key clue.
Before changing anything, confirm: kubectl describe ds shows only /var/run/docker.sock; agent logs on one affected node report no Docker runtime or failed runtime detection; /run/containerd/containerd.sock exists on the node; crictl info shows containerd; and the agent is otherwise healthy. I would also check whether the 12 nodes are mixed-runtime or fully containerd, because that determines whether to keep both sockets.
2. Propose a secure remediation that satisfies the read-only hostPath policy.
Read the worked answer
Add a read-only hostPath volume for /run/containerd/containerd.sock and mount it into the agent container; keep the Docker socket mount only if some nodes still use Docker. This does not require a privileged pod or a writable hostPath, so it fits the stated policy. I would still flag the residual risk: access to a runtime socket is powerful even when the mount is read-only, because the API is callable. If security review treats the socket as too sensitive, the alternative is a restricted socket proxy, but that adds a component and latency.
Make the change in the Helm values/GitOps repo, roll the DaemonSet, and do not restart application pods. Ensure only the correct runtime sensor is active per node to avoid duplicate entities.
3. How would you validate the fix and prevent duplicate or stale container entities?
Read the worked answer
Validate at three levels: node (crictl ps count), agent (container count and no runtime errors), and Instana UI (the 180 containers appear with CPU/memory and correct pod/namespace). Total container count should be about 1,200, not 1,380. Duplicates usually mean both Docker and containerd sensors are reporting the same workload, or stale entities have not aged out. Check that container IDs are unique and that the agent is not configured to force both runtimes.
Watch cardinality for a few days; if duplicate or short-lived containers explode, normalize names or exclude ephemeral namespaces. Finally, confirm capacity dashboards recover for the 12 nodes.