TS-C (NCCT): Agent Rollout Stalls at 8% and the OT Network Has No Internet

Order by blast radius, gate on measured reporting, not on install success.

1. Pilot (500 devices) - already done, but it exposed the 8% silence. Do not widen until the silence is explained; a silent agent looks identical to a compliant host on a dashboard.
2. Windows endpoints, wave 1 (~10%) - expand only after reporting rate in the pilot holds above target for seven consecutive days, including across at least one certificate or policy change.
3. macOS endpoints - separate wave, because MDM packaging fails differently from GPO; do not blend the failure signals.
4. Servers - last, and aligned to the first Saturday maintenance window so any agent-induced performance impact or reboot requirement lands inside an approved outage.
5. OT segment - its own deployment with its own approval chain; see the air-gap task below.

Gates that must pass before each widening: check-in rate above target, no unexplained service restarts, no endpoint performance complaints above baseline, and a tested rollback (GPO uninstall or MDM removal) proven on at least one device per platform.

Do not let a schedule force a wave. The cost of a bad agent build at 25,000 endpoints is far higher than a two-week delay.

Hypothesis ranking, with the certificate rotation as the leading suspect but not the assumption.

1. Trust chain broken on the endpoint. The agent validates the platform certificate. If the new certificate was issued by a CA not in the endpoint trust store, or the intermediate is missing, or the certificate was issued for a name the agent does not use, the agent will fail silently after the old certificate expires. Check endpoint-side agent logs for TLS handshake errors before checking anything on the platform.
2. Rotation timing overlap. If the old certificate was revoked or removed before the new one propagated, agents that had not yet re-enrolled will be stranded. Confirm whether an overlap window existed.
3. TLS inspection or proxy changes pushed in the same change window - a common co-cause when the two events share a date.
4. Clock skew on endpoints, which breaks validation in a way that looks like a certificate problem.
5. Endpoint protection quarantining the agent service after a signature update.

Evidence trail: pull agent logs from a sample of ten silent hosts across two OS families, correlate the silence start time against the change window to the minute, confirm the certificate chain from an affected endpoint with a manual TLS connection, and compare a silent host against a working host of the same build.

Recovery: reissue with a trust chain already present on endpoints, and run rotations with an overlap period where both certificates are valid.

The air gap is a design input, not an obstacle to route around.

Content:

• Use offline content packages transferred on approved media through the plant's existing media-handling procedure, with hash verification and a documented chain of custody.
• Stage the import on a local relay inside the OT segment so the platform never needs a route to the internet, and so the update can be rolled back if plugin content causes instability.
• Version and date every package; an auditor will ask what content was live during the audit period.

Assessment:

• Passive collection first - network flow or span-based discovery to inventory the segment without touching devices.
• Low-impact credentialed checks second - restrict to configuration and patch-level reads, and explicitly exclude plugins known to be disruptive to controllers, historians, and safety systems. Enable the platform's safe-check setting and maintain a documented exclusion list.
• No active network scanning of controllers regardless of how convenient it is.

Governance: get the plant manager's constraints written into the scan policy as technical exclusions, not as verbal agreements. The exclusion list becomes the evidence that you honoured the operating constraint, and it tells the next engineer why the segment looks under-scanned.