When the Numbers Say the Project Worked
A logistics firm deployed a new dispatch system eight months ago. The business case promised a 15 percent reduction in average delivery time and a 20 percent cut in dispatcher overtime. Measured results show delivery time down 4 percent and overtime down 18 percent. The operations director calls the project a success because overtime is nearly at target. The finance analyst notes that overtime fell mainly because two depots closed and headcount dropped by 60 dispatchers, and that the remaining dispatchers are handling 22 percent more deliveries each. Customer complaints about late deliveries rose 30 percent in the same period, but complaints are logged in a separate system the project team never examined. A fuel price increase and a new competitor entering the region also occurred during the period.
1. Evaluate the solution's performance against the business case and identify the measurement errors in the operations director's conclusion.
Read the worked answer
Separating the two measures
The two benefits need to be assessed differently because they behave differently.
Delivery time, down 4 percent against a 15 percent target, is a clear miss on its own terms, and the 30 percent rise in late-delivery complaints points the same direction. Two independent measures agreeing that service did not improve is stronger evidence than either alone. The project team's failure to examine the complaints system is a significant evaluation gap: it had a readily available source of outcome data and did not look at it.
Overtime, down 18 percent against a 20 percent target, looks like near-success but is almost certainly not attributable to the solution. Overtime fell mainly because 60 dispatchers left when two depots closed. Fewer dispatchers means fewer overtime hours available to pay, mechanically. Meanwhile each remaining dispatcher handles 22 percent more deliveries, which is a workload increase, not an efficiency gain. If anything, the overtime reduction is being purchased with higher individual load, and that is a sustainability risk, not a benefit realization.
The attribution errors
Three errors are present. First, the operations director is comparing an outcome to a target without asking what caused the outcome. Second, the analysis ignores the counterfactual: what would overtime have been with 60 fewer staff and no new system? Third, confounders are unaddressed. The fuel price rise plausibly affected delivery times, and the new competitor plausibly affected volume mix and routing priorities. Neither is mentioned in the director's assessment.
What the analyst should do
Reconstruct the measures with a defensible baseline, normalize overtime per delivery or per dispatcher rather than as an absolute, incorporate the complaints data as a service-quality measure, and explicitly document the confounders so the conclusion states what can and cannot be attributed to the solution. The honest current finding is that the solution has not demonstrated its delivery-time benefit and that the overtime figure is not evidence of benefit at all.
2. What should the business analyst recommend regarding the solution's future, and how should the recommendation be governed?
Read the worked answer
The recommendation
The evidence does not support declaring success, and it does not support abandoning the solution either. It supports a limited-life extension with corrective action, because the delivery-time miss is modest and may be addressable, while the overtime finding is an artifact rather than a result.
The analyst should recommend three things. First, continue the solution but re-baseline the benefits with the confounders documented, so future reporting is honest. Second, investigate why delivery time improved only 4 percent: is the dispatch logic underused, are dispatchers overriding its recommendations, or is the constraint elsewhere in the network? The 22 percent workload increase suggests dispatchers may be working around the system under pressure, which would explain both the small gain and the complaint rise. Third, address the service-quality signal directly, since a 30 percent complaint rise is a customer-retention risk that outweighs a partial overtime saving.
Governance
The recommendation should go to the body that approved the business case, not to the operations director alone, because the director has already publicly framed the project as a success and has an interest in that framing. The analyst should present the evidence with the attribution analysis attached, state clearly which benefits are realized, which are unproven, and which are contradicted, and ask the governing body to decide whether to fund the corrective investigation.
The analyst should also recommend that the benefits register be updated to include the complaints measure and to record the depot closures as a contextual change, so the record reflects what actually happened. Finally, the analyst should propose that future benefit reviews include a standing check for confounders and a defined set of outcome sources, so that a whole system of evidence is not left unexamined again.