The Pilot That Wasn't Allowed to Fail
A logistics company is rolling out a new warehouse management system across 12 sites. The plan calls for a pilot at one site before national roll-out. The project manager sets quality criteria for the pilot as 'system operational' and schedules a single acceptance test at the end of the pilot. No quality records are kept. During the pilot, staff work around a barcode-scanning defect by manually keying orders; the pilot is declared a success because orders were dispatched. The defect is not logged. National roll-out begins, and within three weeks the manual workaround cannot cope at 12 sites. The board asks why the pilot did not catch this. The project manager says 'the pilot passed its acceptance test.'
1. Using the quality theme, explain why the pilot failed to detect the defect.
Read the worked answer
Quality planning and quality control were both weak
- Quality criteria were too vague. 'System operational' does not define measurable acceptance criteria (e.g. scan accuracy ≥ 99.5%, orders processed without manual intervention). Vague criteria let a workaround count as success.
- No product descriptions. Each product (the pilot system, the configuration, the training materials) should have a product description with quality criteria, quality method, quality tolerance and quality responsibilities. Without them, no one knew what 'good' looked like.
- No quality register or quality records. The workaround was never recorded, so the defect was invisible. Quality records provide the evidence that quality activities happened and what they found.
- Single end-point acceptance test. Quality control should be applied throughout - checking products as they are produced, not only at the end. A single acceptance test at the end of the pilot is a late, weak control.
- No independent review. The pilot was declared successful by the people who wanted it to succeed. Project Assurance or an independent reviewer should have challenged the workaround.
Judgement
The pilot 'passed' only because the criteria were written to allow a workaround to pass. That is a quality planning failure, not a testing failure. The defect should have been logged as an issue, assessed, and either fixed or formally accepted as a known limitation with a decision by the board.
2. What should the project manager have set up before the pilot began?
Read the worked answer
Before the pilot
- Quality management approach (part of the PID) - what quality activities will be used, who is responsible, what records will be kept.
- Product descriptions for the pilot system, configuration, data migration and training, each with:
- quality criteria (measurable)
- quality method (test, review, inspection)
- quality tolerance
- quality responsibilities (producer, reviewer, approver)
- Quality register - a log of planned and completed quality activities and their results.
- Quality records - evidence of each check, including defects found and their resolution.
- Acceptance criteria agreed with the Senior User - what 'success' means in measurable terms, including performance under load.
- A defined review/acceptance point with named approvers and a clear rule that workarounds must be logged.
Judgement
Quality must be planned in, not inspected in at the end. The pilot was the project's best chance to find the defect cheaply; without product descriptions and quality records, it could not.
3. The defect is now affecting 12 sites. How should it be handled under the issue and change control procedure?
Read the worked answer
Handling the defect
- Capture as an issue in the issue register - it is a problem that is now affecting delivery.
- Assess impact on time, cost, scope, quality, risk and benefits across all 12 sites.
- Determine whether it is a change. Fixing the defect may require a change to the product (configuration or code) and to the plan. If so, raise a change request and follow change control.
- Escalate if beyond tolerance. With 12 sites affected, the impact is likely beyond the project manager's tolerance. Raise an exception report to the board with options: fix before further roll-out, pause roll-out, or accept a workaround with a defined end date.
- Update the risk register - the workaround is now a live risk to service.
- Record lessons - the pilot's failure to log the workaround should be captured as a lesson for future roll-outs.
Judgement
The board must decide, not the project manager alone. The project manager's 'the pilot passed' defence is not a defence: the defect was known in practice but not recorded, which is precisely what issue and quality records exist to prevent.