Deployment
Domain 5: Deployment
Deployment tests whether you can move a model into operational use and keep it working. CAP questions address integration patterns, scoring modes, user acceptance, monitoring, change management, rollback, and handoff.
Core rules and concepts
- Choose the deployment mode from the decision: batch scoring for periodic campaigns, real-time APIs for transaction decisions, streaming for continuous events, or embedded scoring in a business application.
- Define integration requirements: input and output contracts, latency, throughput, availability, security, and failure behavior.
- Plan user acceptance testing (UAT) with business users, including edge cases, threshold behavior, and comparison with the current process.
- Design monitoring before launch: data quality checks, score distributions, performance metrics, drift indicators, and alert thresholds.
- Prepare operations: runbooks, ownership, incident response, rollback plan, versioned artifacts, access controls, and audit logs.
- Manage change: train users, communicate limitations, update policies and workflows, and measure adoption. A model that is not used has no value.
- Validate in a shadow or pilot mode where risk is high, then scale with explicit go and no-go criteria.
Worked example
A fraud model must score card transactions in under 100 ms. The team deploys it as a real-time API behind the authorization service, with a fallback rule if the API times out. They define an input contract for transaction and customer features, precompute static features, and monitor latency, error rate, score distribution, and fraud capture rate. In UAT, business users test threshold changes and false-positive handling. A two-week shadow mode compares model decisions with the existing rule without blocking transactions. Go-live criteria include latency under 100 ms at the 95th percentile, error rate below 0.1%, and no increase in customer friction. A rollback switch restores the prior rule.
Traps that cost marks
- Launching without UAT or a rollback plan.
- Ignoring latency, throughput, or failure behavior in real-time deployment.
- Monitoring only model accuracy and missing data quality, drift, or business outcomes.
- Handing off without documentation, ownership, or runbooks.
- Skipping change management and assuming users will adopt the model.
- Treating deployment as the end rather than the start of monitoring.