AI Solution Lifecycle and Delivery
AI Solution Lifecycle and Delivery
This domain tests how AI solutions move from problem framing to production and iteration. Expect questions on acceptance criteria, baselines, evaluation metrics, human-in-the-loop design, and release management.
Core rules and concepts
- Start with a decision or workflow problem, not a model. Define the user, decision, action, and success metric.
- Establish a simple baseline: rules, current process, or an off-the-shelf model. A complex model must beat the baseline on the business metric.
- Choose evaluation metrics tied to costs: precision and recall trade-offs, calibration, latency, and subgroup performance. Accuracy alone is rarely sufficient.
- Use stage gates: discovery, prototype, pilot, production, and scale. Each gate has entry and exit criteria, including non-functional requirements.
- Design human oversight: review thresholds, escalation paths, feedback capture, and rollback. Shadow mode and canary releases reduce deployment risk.
Worked example
A document-classification solution must route invoices. Baseline: clerks route 1,000 documents per day with 92% accuracy. The model achieves 96% overall accuracy, but 88% on handwritten invoices, which are 15% of volume. The team sets a subgroup threshold of 90%; the model fails. They add a human-review queue for handwritten documents and a feedback loop. Reasoning: overall accuracy hides a costly subgroup gap; the release criterion must reflect the workflow and error costs.
Traps
Optimizing accuracy without a baseline; skipping non-functional requirements such as latency and auditability; launching without monitoring; and treating human review as a permanent patch instead of a designed control.