Nine Months In: Judging a Portal That Has Not Delivered Its Business Case
Nine months after go-live of a new customer service portal, adoption sits at 41% of the target defined in the business case. Average call handling time has risen 12%, manual re-keying persists because two order-entry screens were descoped, and the business case promised a 20% reduction in manual entry. There are 214 open defects, 31 of them severity 1 or 2, concentrated in order amendment and returns. The finance analyst cannot confirm or refute the benefit because no baseline of manual-entry volumes was captured before go-live. The vendor's account manager argues the figures are 'within normal ramp-up' and asks for a decision at the next quarterly review, where the business sponsor also wants approval to expand the portal to two more regions in Q3.
1. How would you measure solution performance here, and what would you do about the missing baseline?
Read the worked answer
Measure against the business case's intent, define each measure so it cannot be argued about later, and be honest about the missing baseline.
Measures to define
- Adoption: distinct active users per week as a percentage of the target population, not logins, plus task completion rate by role.
- Efficiency: average handle time per contact type, first-contact resolution, and manual re-keying events per order - the last is the actual benefit lever in the business case.
- Quality: defect escape rate, rework rate, and the trend in severity 1 and 2 open counts.
- Value: cost per contact, and the 20% manual-entry reduction expressed in hours and currency. For each measure specify definition, formula, data source, frequency, owner, and baseline or target. A measure without a source and an owner is an opinion.
The missing baseline You cannot invent one. Options in order of preference:
- Reconstruct from system logs - order volumes pre- and post-go-live plus contact volumes may allow an indirect estimate of re-keying.
- Use a control comparison - the two regions not yet migrated give a natural comparator; measure their manual-entry rate now.
- If neither is credible, re-baseline forward and state that the original 20% claim is now measured against a revised figure, reporting the variance to the sponsor and finance explicitly rather than presenting a number you cannot defend.
Why this matters The quarterly decision depends on whether the shortfall is performance or measurement. Separating those is the analytical task, and the sponsor is entitled to a clear statement of which it is.
2. How would you determine whether the shortfall is caused by the solution, by the enterprise, or by an overstated benefit case?
Read the worked answer
Establish cause before recommending anything, because each cause implies a different decision.
Solution limitations
- Two order-entry screens were never built, so the manual re-keying benefit is structurally undeliverable regardless of user behaviour. That alone explains part of the gap.
- 214 open defects: triage by business impact rather than by count. Which of the 31 severe defects sit on the five highest-volume contact types?
- Handle time up 12%: test whether the increase concentrates on affected contact types (a solution cause) or is uniform across all contacts (an environmental cause such as volume or staffing).
Enterprise limitations
- Training and change management: did users receive role-based training, and is there a coaching mechanism?
- Process alignment: are staff still required to update the legacy system as well? Dual entry would fully explain persistent re-keying.
- Upstream data quality, organisational policy, and incentives - if handle time is not measured for the new process, behaviour will not change.
Method Segment performance by region, role, tenure and contact type; compare adopters with non-adopters; interview and observe eight to ten non-adopters doing the task. Observation is more reliable than asking people why they do not use something.
The claim to test Was the 20% manual-entry reduction ever achievable within the scope actually delivered? If the business case assumed screens that were descoped, the benefit case was misstated. That belongs in the assessment, not in a narrative about ramp-up.
3. What recommendation would you take to the quarterly review, and what evidence and conditions should support it?
Read the worked answer
Give the review a recommendation with evidence, conditions, and a decision the sponsor can own.
On 'within normal ramp-up' Ask for the ramp curve to be produced as an approved assumption with dates and a defined plateau. If it was never agreed, it is a post-hoc claim. Nine months with adoption at 41% of target, plus a structural gap from missing screens, does not support it.
Options and the reasoning for each
- Continue with remediation - justified only if the shortfall is predominantly transition-related and the structural gaps are cheap to close. Conditions: build the two screens, targeted coaching, a defect burn-down target, weekly adoption reporting against a defined curve, and review at 90 days.
- Adjust the solution - requirements were incomplete rather than the concept being wrong, and the benefit case is re-baselined to delivered scope.
- Partial retirement or replacement - where observation shows the portal adds work for the main contact types with no offsetting benefit.
- Delay expansion - expanding to two more regions before the current deployment demonstrates value multiplies the loss and destroys any ability to isolate cause.
Recommended position Do not expand in Q3. Approve a 90-day remediation with named actions, funding for the two screens, a benefit target re-baselined with finance, and a decision gate with explicit exit criteria (for example, adoption at 70% of the revised target and re-keying down 15%). Recommend retirement for any module where evidence shows no achievable benefit.
State the uncertainty plainly, including the limits imposed by the missing baseline, and record the decision, rationale and conditions.