TS-C (NCCT): The Fix That Drift Enforcement Will Revert at 02:00

1. Suspend enforcement for those three nodes (do this before anything else). If 02:00 arrives, the tool reverts the fix, the partner integration breaks again, and it breaks at 02:00 on a Sunday with nobody watching and no incident open. Suppress the override, not the whole fleet, and put an expiry on the suppression so it cannot be forgotten.

2. Capture the live state. Dump the exact TLS configuration, the full cipher list, timestamps, and the node names. Once enforcement runs or anyone edits those files, the only record of what actually worked is gone - and "what actually worked" is the artefact you need to codify. Do this before touching anything else.

3. Confirm causation before codifying. Pull the partner's handshake-failure logs, the incident timeline, and test a staging node against a client that behaves like theirs. It is entirely possible the handshake failures stopped for a different reason at the same minute. Codifying a coincidence into the base configuration puts an unnecessary TLS relaxation into production for all twelve nodes.

Why the order matters

Step 2 after step 3 risks losing the evidence; step 1 after step 2 risks losing the configuration. Step 1 is cheapest and most urgent, so it goes first. The irreversible action is the one you sequence around.

Classification: emergency change, approved retroactively. Justification: service-impacting incident, revenue-affecting partner integration, fix applied under time pressure, no pre-approval possible. Emergency classification is legitimate - the failure was not applying the fix, it was never recording it. Retroactive approval must still be real: a named approver reading the record after the fact.

Required fields

• What changed: TLS minimum version and cipher list, stated precisely, with the before and after values.
• Why: the partner handshake failure, with the incident link INC-4471.
• Blast radius: 3 of 12 nodes; partner traffic is 4% of revenue; no other consumers affected - or, if other clients also negotiate TLS with those nodes, that is the gap to check before accepting.
• Verification evidence: handshake success from the partner, error-rate and latency before/after.
• Rollback: how to revert, and what breaks if you do.
• Who applied it and who approved it, with timestamps.
• Follow-up action and owner: codify into the repository, or retire the change once the partner upgrades.

Security note. Relaxing TLS minimum version has a compliance dimension. The record should say why the weaker setting is acceptable, its expiry condition, and who owns retiring it. Without that, the exception becomes permanent by default.

A break-glass path that intercepts interactive changes. The real gap is not detection speed - a daily scan would have flagged the drift on Monday, which is still five days late and still a surprise. The gap is that nothing stood between the engineer's SSH session and the machine.

Design

• Interactive administrative changes go through a documented override that auto-opens a change record and creates a bounded enforcement exemption with an expiry, defaulting to something short like 24 hours.
• The exemption carries the reason, the incident link, and the owner, and it expires automatically rather than relying on somebody remembering.
• If no override is opened, the next enforcement run still reverts - that behaviour is correct and should stay, because it is what makes the repository the source of truth.

Secondary improvements

• Enforce mode should be paired with an exemption mechanism. Enforcement without a sanctioned escape hatch does not prevent drift, it just teaches engineers to disable enforcement when it is inconvenient - which is a worse outcome than this weekend's near-miss.
• Move the drift scan from weekly to daily, not for this incident but for the next one.
• Pipeline-level validation: a TLS profile change should fail review unless it names the client compatibility justification and an expiry. The change would then be reviewed by someone who knows the partner contract exists.