Operations, Monitoring, and Lifecycle Management
What this domain tests
Operations, Monitoring, and Lifecycle Management tests how an agent is deployed, versioned, evaluated, observed, and retired. Expect questions on release control, telemetry, cost, drift, and incident response.
Core rules
- Version every artifact: prompt, model, tool schema, retrieval index, and policy. Pin model versions for production and record the combination as a release.
- Evaluate before release with a fixed test set that includes normal, edge, and adversarial cases. Track task success, groundedness, tool-call accuracy, latency, and cost per task.
- Use canary or staged rollout with a rollback path. A change to a prompt or index is a production change and needs the same control as code.
- Monitor more than uptime: tool error rates, retrieval hit rate, escalation rate, refusal rate, token spend, and user feedback. Set alarms on leading indicators.
- Retire data and agents on a schedule. Remove stale indexes, rotate secrets, and keep audit records according to policy.
Worked example
A procurement agent moves from version 1 to version 2 with a new supplier-search tool. The team runs the evaluation set, deploys v2 to 10% of traffic, and watches tool error rate and escalation rate against v1. After 48 hours with no regression, traffic moves to 100%. If the error rate crosses the alarm threshold, routing returns to v1 and the index snapshot is restored. The reasoning is that the tool change affects action accuracy, so the release is gated by behavior metrics, not just availability.
Traps
Do not monitor only model latency; a fast wrong answer is still a failure. Do not change a prompt directly in production without a version and rollback. Do not ignore cost per successful task. Do not delete audit logs when retiring an agent.