TLS Retirement Across a 400-Node Fleet

The drifted nodes come first because they destroy your understanding of the blast radius. You cannot predict the effect of a cipher change on 37 hosts whose current cipher list is unknown. Worse, the usual rollback - republish from the repository - would silently overwrite someone's six-month-old fix, which may have been a legitimate workaround for a partner or a kernel limitation.

Sequence:

1. Reconcile drift. Capture each drifted node's live config, diff against the repository, and either fold the change in as a reviewed commit or revert it with owner sign-off. No rollout until the fleet state is known and reproducible.
2. Check the agent's own dependencies. Confirm the management agent does not itself speak TLS 1.0 to the server. Turning 1.0 off before upgrading agents is the classic way to lock yourself out of 400 machines.
3. Split the change. Certificate rotation and protocol disablement are separate tickets even if they ship in one window. One variable at a time means a failure is attributable.
4. Canary 5 nodes: rotate the cert with the protocol set untouched, verify handshakes from every partner endpoint, then disable 1.0/1.1 on the canary only.
5. Wait a full cycle plus the longest partner interval. With a 30-minute pull and a nightly partner batch, an hour of green proves almost nothing. Observe overnight.
6. Ramp region by region with a pre-agreed abort criterion, e.g. handshake failure rate above 0.1% or any partner batch failure.

Get facts per partner first: which endpoint, which client library and version, whether an upgrade is feasible, and their change lead time. Then act per partner, not per policy.

• Negotiate a dated deprecation against the 30-day requirement, with a named owner on their side.
• Where 30 days is impossible, offer a documented, time-boxed exception with compensating controls: a separate listener on a distinct port bound to their source IP range, mutual TLS with a pinned certificate, network segmentation, and elevated logging. The exposure is isolated to that path rather than the whole fleet.
• Record the exception as risk acceptance with an expiry date and a review. A config that no one owns becomes permanent.

What I would refuse: leaving 1.0 enabled globally so one partner's nightly 4% keeps working. That re-opens the vulnerability on 397 nodes to spare three endpoints. I would equally refuse to simply turn it off without written notice and a tested fallback - a failed nightly transfer has downstream consequences the security finding never mentions, and the fallout lands on whoever flipped the switch.

Rollback must be a state you publish, not a command you type.

• Latency reality: with a 30-minute pull and no push channel, an abort takes up to 30 minutes per node to take effect, and a node that misses its window keeps the bad config until the next one. That caps your blast radius at roughly one canary group or one region - never evaluate an abort criterion mid-fleet.
• Config rollback: revert the config object to the last known-good revision, publish it as a new version (do not rewrite history), and confirm every node reports that revision hash in its inventory. Fleet-wide revision-hash agreement is the definition of 'rolled back'.
• Certificate rollback is separate and slower. Keep the previous certificate and key on disk with correct permissions and a known expiry until the new one has survived a full cycle. Re-issuing takes longer than re-pointing. Before you start, verify the old certificate's remaining validity is long enough that rollback is possible - if it expires during the window you have no rollback and should not proceed.
• Also verify that the agent will still fetch config if the listener it uses is misconfigured; otherwise the rollback channel is the first thing you lose.