Solution Evaluation
Solution Evaluation assesses whether a delivered, partial or proposed solution actually delivers the value expected, and what to do when it does not. The exam tests whether you can measure performance against defined targets, separate solution limitations from enterprise limitations, and recommend an action - including retiring the solution.
The five tasks
- Measure Solution Performance - define and collect measures tied to the business objectives the solution was meant to serve, not just technical uptime.
- Analyze Performance Measures - compare against baselines and targets; look for trends, segments and root causes rather than averages.
- Assess Solution Limitations - defects, gaps and constraints inside the solution that cap performance.
- Assess Enterprise Limitations - factors outside the solution: culture, policies, structure, skills, incentives, processes and other systems.
- Recommend Actions to Increase Solution Value - adjust, extend, replace or retire, each with expected value and cost.
Rules that decide questions
- A measure is only useful with a baseline and a target defined before evaluation; measuring after the fact invites rationalisation.
- Root cause comes before remedy: low adoption may be a training gap, an incentive misalignment or a genuine usability defect - each has a different treatment.
- Enterprise limitations are the ones most often missed, because they sit outside the solution boundary and often outside the team’s control.
Worked example
A CRM rollout targets 80% field-staff usage within a quarter; actual usage is 55%. Analysing by region shows two regions at 85% and one at 20%. Users in the low region log in, then re-enter the same data into a legacy scheduling tool. Assessment: the solution limitation is a missing integration with that tool; the enterprise limitation is a regional incentive scheme that pays on scheduling-tool records, plus no training rollout in that region. Recommendation: prioritise the integration, align the incentive with CRM records, and run targeted training - with an expected recovery to 75% next quarter, tracked monthly. Doing nothing is documented and rejected because the target is contractual.
Traps that cost marks
- Reporting usage averages and missing the segment that explains the number.
- Blaming the solution for enterprise limitations, or the reverse.
- Measuring only what the system reports easily (logins) instead of value measures (cycle time, error rate).
- Recommending “more training” without evidence of a skills gap.
- Never recommending retirement - end-of-life is a valid, examinable action.