Corvus Freight's 100% On-Time Illusion
Corvus Freight runs its finance, warehouse and customs systems on a single ERP platform. Deployments happen four times a year in 'deployment weekends', each touching 6,000 users across 40 depots. The CIO's dashboard reports '100% on-time deployment' because every go-live has started within the scheduled window. In practice, three of the last four deployments had partial failures: customs interfaces were down for two days, 40 users could not post goods receipts for a week, and two depots reverted to spreadsheets. None were logged as deployment failures. The service desk sees a 300% ticket spike for ten days after each deployment, but has no baseline. There are three deployment coordinators, no post-implementation review, and no agreed definition of deployment success. The next deployment is in eight weeks, and an unresolved customs interface defect is still open.
1. Critique the '100% on-time deployment' metric and define a balanced set of measures with precise definitions, data sources and owners.
Read the worked answer
Critique: 'on-time deployment' measures whether the go-live event started in the window. It says nothing about whether the service worked afterwards, so three partial failures were invisible. It is a single, easily satisfied measure that rewards declaring success at the start rather than at verified handover. Better set, with definitions:
- Deployment success rate: deployments meeting all exit criteria within the agreed window - interfaces live, critical transactions tested, no Severity 1 or 2 incident in 72 hours, handover accepted. Partial failures count as failures.
- Change failure rate: percentage of deployments causing rollback, emergency fix or customer-visible incident within seven days.
- Deployment lead time: from approved release candidate to verified production use, split into execution and wait time.
- MTTR for deployment-caused incidents: detection to service restoration.
- Post-deployment ticket rate: tickets per 1,000 users per day compared with a pre-deployment baseline, for ten days.
- Outcome/adoption: percentage of intended users completing key transactions within five days. Owners and sources: the deployment manager owns success and exit criteria; the service desk owns the ticket baseline; incident management owns failure classification; finance and operations own adoption. Data comes from deployment records, monitoring, ITSM and depot reports. Agree definitions before measuring, and report the whole set - speed alone will mislead.
2. Design a continual improvement approach: how to baseline, review deployments and feed lessons back. Include how to avoid gaming and how partial failures should be recorded.
Read the worked answer
Baseline first: reconstruct the last four deployments from records, tickets and depot reports. Classify each as full success, partial failure or failure using the new definitions, including duration and business impact. This provides a credible starting point and exposes that the previous figure was a definitional artefact. Post-implementation review: within 10 working days, blameless, with deployment, application, service desk, depot representation and the business owner. Agenda: intended outcome, what happened, exit criteria met or missed, causes, and actions with owners and dates. Track actions to closure at the next review. Feedback loop: route systemic causes into design, build and release readiness - for example, interface testing must include a live customs message exchange, not just a test harness. Add unresolved defects to go/no-go criteria. Continual improvement cadence: monthly metric review and a quarterly deep dive with the CIO. Apply the continual improvement model: vision, where we are now, where we want to be, how to get there, take action, check progress. Avoiding gaming: do not turn one metric into a target; audit a sample of 'successful' deployments; record partial failures explicitly with severity; separate reporting from judgement; publish trends, not just the latest value. Reward early honest disclosure of defects, otherwise the interface problem will surface again after go-live. Set targets from the baseline - for example, halve the ticket-spike duration within two deployments.
3. Eight weeks before go-live, the customs interface defect is unresolved. Set out your go/no-go criteria and decision, using the metrics to inform it, and state what to do if the criteria are not met.
Read the worked answer
Judgement: the unresolved customs interface defect directly threatens a critical business transaction and has precedent - customs interfaces were down for two days after a previous deployment. Starting would repeat a known failure mode. Criteria for proceeding: confirmed root cause and tested fix; end-to-end customs message testing passed in a production-like environment with the customs authority or broker; a rollback or fallback exists (manual declaration process) with trained staff; exit criteria explicitly include a customs transaction test and 72-hour stability; the business owner accepts residual risk in writing; service desk and depots are briefed. Using metrics: the baseline ticket rate and prior interface outage duration give an expected impact range, which should be presented to the business owner rather than a binary opinion. A go decision requires that the predicted impact is understood and accepted, not merely that the window is available. If criteria are not met: defer the affected scope, deploying independent ERP changes while keeping customs on the current version, or postpone the weekend. Partial scope is legitimate only if the interface can stay on the old version without data inconsistency; verify that before choosing it. Record: document the decision and rationale. If deployment proceeds and fails, pre-agreed exit criteria and fallback speed the response and make the review honest. The metric set is only useful if it changes this decision.