Agent Upgrade and OpenTelemetry Cutover Across Three Clusters

Order by blast radius on detection, not by tidiness.

First, stop the double counting on the 40 dual-reporting services. Decide the target path per service and remove the redundant agent, or tag both sources and filter one out of every dashboard and alert query. Doubled throughput distorts any capacity judgement and will corrupt the upgrade validation.

Second, fix alert scoping. Convert the 18 name-based policies to tag or entity-GUID scoping so OTel services stop reporting no-data and host-name coupling disappears. This is the highest-value hour of work in the whole plan because it restores detection coverage.

Third, run the mandated upgrade in waves with fleet control: canary on roughly 5% of nodes in one cluster, validate metric and trace parity against pre-upgrade baselines, then proceed cluster by cluster, leaving buffer before the deadline. Do not upgrade while instrumentation ownership is ambiguous on a service, because a failure will be unattributable.

Last, repair the golden-signal dashboard against the OTel metric names and resource attributes. Dashboards lag detection; a blank panel is an inconvenience, a missed outage is not. If the deadline slips, negotiate scope on the upgrade, not on alert coverage.

Separate two questions: is the signal bad, and is the signal missing.

For threshold alerting, scope policies to tags or entity identifiers derived from the OTel resource attributes, principally the service name, rather than to hard-coded entity names. Confirm that entity synthesis is actually creating entities for the OTel services; if the collector export is missing the attributes the platform uses to synthesize a service entity, no policy scoping will help and the gap must be fixed at the collector's resource processor.

For no-data, treat loss of telemetry as its own alert with its own owner, evaluated on a heartbeat or a minimum expected data rate, and do not let it share a policy with threshold breaches. A service that silently stops reporting is indistinguishable from a healthy one under a threshold-only design, which is exactly how the 18 open policies became invisible.

Then validate: deliberately stop one OTel service in a non-production namespace and confirm both the threshold policy and the heartbeat alert behave as designed. Finally, remove host-name scoping everywhere and audit what remains, since the 14-minute late detection came from the same root cause.

Split the fleet and be explicit about the residual risk.

Use fleet control for the 60% it can reach, running the canary-then-wave pattern inside that population. For the remaining 40%, fall back to the existing configuration management path, a baked node image, or a DaemonSet rollout if the agents are containerized, and track them on a compliance dashboard by node so nothing is silently forgotten.

In parallel, fix the privilege gap at its source: define the least-privilege policy that fleet control requires, get it approved once, and apply it through the cluster role model so future upgrades do not repeat this. Where a team refuses the privilege, escalate the exception with the compliance owner rather than accepting a quiet gap, since the upgrade is mandated.

The judgement call is what happens near the deadline. If the unreachable set is small and non-critical, document a time-boxed exception with a compensating control such as a read-only health check per node. If it includes production nodes running critical services, the exception is not acceptable and the privilege work becomes the priority, ahead of dashboard repair.