Solution Evaluation: Measuring Delivered Value and Finding Constraints
Solution Evaluation asks whether a deployed or partially deployed solution delivers the value the enterprise expected, and if not, why. BABOK Guide v3 defines five tasks: Measure Solution Performance, Analyze Performance Measures, Assess Solution Limitations, Assess Enterprise Limitations, and Recommend Actions to Increase Solution Value. Expect questions that separate solution limitations from enterprise limitations.
Core rules
- Measure solution performance against measures derived from business objectives, and compare to a baseline captured before or during deployment. Without a baseline, a number proves nothing.
- Analyse performance measures by identifying variances from expected results, testing whether the measure truly reflects the objective, and looking for causes rather than symptoms. Correlation is not causation.
- Solution limitations are factors inside the solution: defects, slow performance, missing capability, poor usability, design constraints.
- Enterprise limitations are factors outside the solution: culture, skills, training, policy, process, incentives, and dependencies on other systems.
- Recommendations can include fixing the solution, changing the enterprise, expanding scope, retiring or replacing the solution, or taking no action. Recommendations follow analysis, never precede it.
Worked example
A new CRM reaches 45% usage against an 80% target, with satisfaction at 3.1 out of 5. Analysis shows usage is concentrated on desktop; mobile page load takes twelve seconds (a solution limitation), and records for a major client segment were never migrated, so staff cannot rely on the data (an enterprise limitation). The recommendation is to fix mobile performance, complete migration, and retrain - not to replace the CRM. The reasoning: two distinct limitation types require two distinct fixes, and replacement would not address either.
Traps that cost marks
- Recommending replacement or retirement before root cause analysis of performance measures.
- Blaming the solution for an enterprise limitation such as missing training or misaligned incentives.
- Reporting raw metrics with no baseline, target, or business objective attached.
- Measuring only lagging indicators and declaring success before behaviour stabilises.
- Stopping at measurement and never producing recommended actions, which is the final task output.