The On-Time Delivery Illusion

The 92% is a weighted average, so it can rise while the segment that received the intervention gets worse. Enterprise is 40% of orders and fell six points, from 94% to 88%. That is a material deterioration, and it is the only segment where the routing software went live. Mid-market stayed flat at 93%. Small business rose 19 points, but part of that gain is an artifact: a low-volume region was reclassified out of the segment, which changes both the denominator and the mix. The group increase from 89% to 92% is therefore consistent with a composition effect plus a definition effect, not with the software working.

The definition change makes the comparison worse. Moving the grace period from two hours to four hours makes every segment look better on paper. Enterprise's 88% is measured more leniently than Q2's 94%, so underlying enterprise performance has fallen by more than six points in like-for-like terms. The headline is not evidence of success; it is a blend of a deteriorating core segment, a reclassification, and a looser threshold. A causal claim about the routing software is unsupported because the software did not affect mid-market or small business, and the group figure cannot isolate its effect.

Build a variance bridge from Q2 89% to Q3 92% with three components: definition change, segment mix/reclassification, and within-segment performance. Restate both quarters under one definition, preferably the two-hour grace period, and disclose the four-hour version as a sensitivity. Show monthly enterprise OTD before and after 1 August, with order volumes, so the board can see the trend rather than a quarterly average. Use mid-market as a comparison group to separate seasonality from the software effect. Check the reclassified region's historical OTD; if it was low, the small-business gain is partly definitional.

Quantify customer impact: number of late orders, penalty payments, complaints, and any service-level credits. A percentage alone hides whether 8% late means 800 or 80,000 orders.

Present the Q3 performance as: 'The headline improved, but the improvement is not attributable to routing.' Use a small table by segment with order share, Q2 OTD under both definitions, Q3 OTD under both definitions, and change. State limitations and confidence. Recommend that the board not cite the 92% as evidence of software success. If the board pack has already gone out, request a correction note before the meeting rather than waiting for the minutes.

Create a metric dictionary for OTD: numerator, denominator, grace period, exclusions, segment definitions, data source, owner, and review cadence. Any change to a KPI definition should go through change control, be logged, and trigger a restatement of history for at least four quarters. A definition change must be flagged in the board pack, not buried in an appendix.

Publish segment-level data alongside any group headline. A rule such as 'no headline without a stratification check' would have caught this. Add data quality triggers: if more than 5% of order volume is reassigned between segments, require a review before the metric is reported. Assign a metrics owner independent of the programme being measured, so the routing team does not own the evidence of its own success.

Require a variance bridge for any material movement in a headline KPI. Separate correlation from attribution: the COO's claim is causal, so it needs an evaluation design, such as a staged rollout or a comparison group, not a before-and-after group average. Define success measures for the routing software in advance: enterprise OTD, late-order cost, route adherence, and customer complaints. Baseline them and review monthly. Audit a sample of OTD calculations quarterly. Culturally, reward accuracy over a green number; otherwise the next reclassification will be just as tempting.