Freeze exception for an expiring TLS certificate
Northwind Freight runs 24/7 parcel sorting across three hubs. Its change process is a single weekly Change Advisory Board (CAB) plus an emergency route that any duty manager can approve. In the last quarter, 312 changes were raised: 121 went through the emergency route, and 38 of those were later reclassified as pre-planned work that had missed the CAB deadline. Internal audit found 14 changes with no recorded test evidence and 6 with no back-out plan. A peak-season freeze begins in six weeks and runs for nine weeks; during that window only security patches and incident fixes are permitted. A critical TLS certificate expiry across two hub scanners is known: renewal must happen in five weeks, but the supplier can only deliver the update during the freeze window. The head of operations says 'freeze means freeze'; the security lead says certificate failure would stop scanning at two hubs. You are the change manager.
1. How should the TLS certificate change be handled? State the decision you would take, who authorises it, and what you record.
Read the worked answer
Decision: treat it as a normal change with an expedited assessment and a time-bound freeze exception - not an emergency change raised on the day.
Reasoning:
- The certificate expiry is known and dated. Emergency change exists for restoring service or urgent risk mitigation where there is no time to plan. Using it here would repeat the pattern that produced 121 emergency changes out of 312.
- 'Freeze means freeze' is also wrong as stated. A freeze controls discretionary risk during a period of elevated business risk. Letting a certificate expire is not zero risk; it is an unmanaged risk with a known date. The freeze should be applied as 'no change unless the risk of not changing exceeds the risk of changing', which is precisely the test the change authority exists to apply.
- So: raise a normal change now, with the change authority and not a duty manager approving an exception. Record the business justification, the residual risk if implementation fails, supplier test evidence, the back-out (rolling back to the old certificate only if still valid - otherwise a documented manual fallback at the two hubs), the implementation window at the quietest sorting period, verification steps, and a named on-call owner.
- Have the security lead and both hub operations managers sign the risk statement, and escalate the exception to the change authority chair in writing. If approved, schedule the earliest possible window, not the last day before expiry.
- Run a post-implementation review and feed it into the audit finding. The real failure is that a known expiry was allowed to become a freeze-window decision.
2. What does the 38 reclassified emergency changes tell you, and what would you change in the process?
Read the worked answer
121 of 312 (39%) is not an emergency rate; it is a second, ungoverned change system running alongside the first. Crucially, 38 of those were pre-planned work that missed the CAB - that is a latency and capacity problem, not a discipline problem. The 14 changes with no test evidence and 6 with no back-out plan are the downstream consequence: when the normal route is slow, people route around it and the evidence goes with them.
What I would change:
- Define standard changes. Pre-authorise low-risk, repeatable, reversible work with a named owner and an audit trail: routine patching, certificate renewals with a known procedure, capacity changes within a threshold. For a portfolio of this size, that is probably the largest single category.
- Delegate normal changes by category to change owners who understand the service. Keep the CAB for high-risk, cross-service and first-of-type changes. A board that reviews everything reviews nothing properly.
- Tighten rather than close the emergency route: written entry criteria, named approver, retrospective review within five working days, and a monthly reclassification metric. Closing it entirely converts emergency change into shadow change.
- Measure lead time, reclassification rate and failed-change rate, and publish them. The 38 reclassifications are the baseline.
Success is not fewer changes; it is fewer unplanned changes and fewer failures.
3. Which two measures would you use to demonstrate the change process is improving, and why would you reject the obvious alternatives?
Read the worked answer
I would use a pair, because either one alone can be gamed:
- Emergency-change eligibility rate - the proportion of emergency changes where the entry criteria were met at the time of raising, not the raw count. This measures whether the route is being used as designed. Counting emergency changes alone rewards suppressing them or relabelling them.
- Failed-change rate on service-critical configuration items - changes rolled back or directly causing an incident, segmented by risk class. This measures whether any speed gained was bought with risk. Aggregate failure rates hide the fact that failures concentrate in a few CIs, as the pricing service example shows.
Rejected alternatives:
- Total change volume - it can fall simply because people stop reporting, and it says nothing about outcome.
- CAB attendance or number of changes reviewed - measures activity, not decisions; a busy board may be the problem.
- Percentage of changes with a back-out plan recorded - a tick-box that is satisfied by a template line. Pair it with evidence that the back-out was tested, if you use it at all.
I would publish both measures monthly with the lead time for normal changes, so the organisation can see the trade-off rather than a single score. If lead time falls and the failed-change rate holds, the redesign is working; if failure rises, the delegation went too far or the standard-change criteria are too loose.