99.2% success and a doubling of change-related incidents
Corvus Financial reports a change success rate of 99.2% to its executive committee each month, and the change manager's bonus is linked to it. Over the same period, major incidents where change was a contributing cause have doubled. Failures are logged as 'defects' and excluded from the change success calculation. CAB performance is reported as 'changes approved per meeting' against a target of 95% CAB utilisation. Four months ago a new change portal replaced email-based RFCs; adoption sits at 30% and is flat. Most teams still email the change manager directly, who registers the requests for them because it is quicker than arguing.
1. Critique this measurement set. What is it incentivising, and what is it not telling you?
Read the worked answer
99.2% success rate is unfalsifiable. The denominator is self-selected: teams choose how to classify a failure, and 'defect' is a label rather than a different kind of event. Any measure whose denominator is defined by the people being measured will be optimised at the label, not at the outcome. Tying the bonus to it converts that from a likelihood into a certainty.
'Changes approved per meeting' measures activity, not value. Approval is not an outcome, and maximising approvals per meeting is in direct tension with adequate assessment. It rewards deciding faster, regardless of whether the decision was right.
'95% CAB utilisation' actively rewards queueing. A full agenda looks like efficiency; an 11-day lead time is invisible in that number. The measure is upside down.
What is missing: change-related incident rate, lead time by risk class, the ratio of emergency to normal changes, volume of unauthorized or shadow change, proportion of work handled as standard change, rework rate, and whether changes achieved the outcome they were raised for. The doubling of change-caused major incidents is the one number that reflects reality, and it is being reported somewhere else entirely.
2. Propose a balanced set of change enablement measures with clear definitions, and say how you would prevent each one from being gamed.
Read the worked answer
Proposed set:
- Change-related incident rate: incidents where a change is a contributing cause ÷ all changes implemented in the period. Denominator sourced from pipeline and CMDB records, not from the change record system, so relabelling cannot remove an event.
- Lead time by risk class: submission to authorization - report median and 90th percentile as a distribution, never an average.
- Emergency-to-normal ratio: trend, not a point value. A rising ratio signals the normal path has become too slow.
- Proportion of changes implemented as pre-authorized standard change: a proxy for whether low-risk work is being handled proportionately.
- Intake completeness at first submission: leading indicator for deferral and rework.
- Unauthorized change count and time to detection: the honest measure of whether the process is being used.
- Outcome achieved at post-implementation review: did the change deliver what it was raised for.
Anti-gaming rules: define each measure's data source outside the team being measured; publish distributions and name outliers rather than a single headline figure; pair every efficiency measure with a quality measure so improvement at one's expense is visible; remove the bonus link to any self-reported metric; and retire measures quarterly once they stop discriminating. A measure nobody can fail is not a measure.
3. Portal adoption has stalled at 30%. Apply organisational change management reasoning: what is the real reason teams email, and what would you actually do?
Read the worked answer
The real reason is that email is faster for the requester. The portal adds a step and delivers its benefit - reporting, visibility, auditability - to the change manager, while giving the team nothing in return. The 30% who adopted are those under the most compliance pressure or with the most to gain.
And the workaround is subsidised. The change manager registering requests on the team's behalf removes the last reason to switch. No amount of training competes with a colleague doing the data entry for you.
What to do:
- Remove the choice instead of demanding it. Generate change records automatically from the pipeline and from the email channel, so a team that emails still ends up with a record and never has to decide. Then close the manual back-door registration by the change manager, with a named exception route.
- Make the benefit personal and visible: approvals in hours instead of days, no chasing, a view of their own change queue and status. That is the case for switching, and it is the only one that will hold.
- Pilot one team, measure lead time before and after, publish the numbers. Peer evidence beats a corporate mandate.
- Enlist team leads as champions rather than broadcasting a message from the change function.
- Set a date and hold it. Sustaining the dual path indefinitely guarantees non-adoption - the portal stays optional and the email habit stays safe.
Training is last-mile support, not the cause. The cause is that the new tool asks for effort and delays the benefit.