02:14 during the change freeze

The engineer restored service to 30% of card transactions at 02:14. That is the correct outcome, and the finding is aimed at the wrong target.

What the process tells us: an emergency procedure that requires two specific named individuals, with no alternates and no out-of-hours coverage, has a single point of failure by design. It will fail precisely when it is needed, because emergencies do not schedule themselves around annual leave or switched-off phones. The freeze is also being applied as an absolute prohibition rather than a risk posture with a defined exception route.

What the finding would achieve: punishing the person who acted in the spirit of the practice teaches the organisation to make the next emergency change silently. That destroys the visibility change enablement exists to provide, and it is a much larger risk than the one being managed.

The defensible position: record the change as unauthorized - that is factually true and the record should be accurate. Then direct the corrective action at the process: no 24/7 authority path, no alternates, no freeze exception rule. The engineer's only realistic alternatives were to wait, or to act. Waiting was not an option anyone would defend at 09:00 the next morning.

Emergency authority at 02:00:

• The on-call role holds standing delegated authority to restore service using documented remediation playbooks - no individual approval needed. The playbook is the authorization; the execution is logged automatically.
• Anything outside playbook scope (schema change, data migration, security configuration) still needs a human authority. Provide a pager rotation of at least three named authorities plus alternates, reachable 24/7, with a published response SLA.
• If no authority can be reached within that SLA, the incident manager holds documented emergency authority for the duration of the incident, with retrospective review. Someone must always be able to say yes.
• Retrospective approval within one business day, recorded against the change.

Freeze rule:

• Default deny, stated as a risk posture rather than a prohibition.
• A written freeze exception route: service restoration or regulatory obligation requires service owner (or delegate) plus business owner. When the alternative is customer-facing outage, the decision may be taken first and ratified after.
• The freeze prohibits non-urgent change. It must never prohibit recovery.

The test of the design is simple: could it be executed at 02:14 on 12 December by one person with a phone? If not, it will be bypassed again.

Must happen:

• Register the change retrospectively with an accurate timeline and the authority that was actually exercised, so the record reflects reality rather than becoming a fiction.
• Post-implementation review within five business days covering: why the load balancer limit was outgrown (a capacity and performance management input, not a change process issue); whether monitoring should have surfaced the approach to the limit earlier; and whether the fix should be turned into a playbook step or a standard change so the next occurrence is pre-authorized.
• Update the playbook and any known error record; feed the capacity finding to the owning practice.
• Amend the emergency procedure: alternates, pager rotation, out-of-hours coverage, freeze exception route.

Must not happen:

• Individual disciplinary action against the engineer. If there is a finding to raise, it belongs against the process design and its owner.
• A review that concludes 'follow the process' without changing the process. That conclusion is only reachable by ignoring that the process had no usable path at the moment it was needed.

The distinction matters because closure is where the organisation decides whether the process is something people use or something people hide from.