Northwind: Operating a Live Agent Without ALM, Capacity, or a Test Set
Northwind's Copilot Studio agent for order status and returns went live six weeks ago for 600 internal users. Resolution rate is 41%, and 28% of escalations route to the wrong queue. Makers edit the live agent directly in production because 'testing takes too long', and no regression test set exists. In month one the tenant's Copilot Studio message capacity was exhausted by day 22, so the agent returned errors for a week. The team now wants to add Azure AI Search grounding over a product catalogue and an action that writes returns to Dataverse. The team has two makers, one administrator, and no documented change approval process.
1. Design the minimum viable ALM process for this agent. What must be true before the catalogue grounding and Dataverse write action are allowed into production?
Read the worked answer
Three environments with security groups: development for authoring, test for verification, production locked to the administrator's deployment role so makers cannot edit live. Authors work in an unmanaged solution; production receives a managed solution through a pipeline with an approval gate. Connection references and environment variables carry endpoints so the same solution moves without manual rework, and the solution is in source control so every release is diffable.
Build the regression set now, because the new features cannot be assessed without it: 150 real conversations from the last six weeks, tagged by intent and expected outcome, including the escalation routing cases that currently fail 28% of the time. Every release runs the set and reports resolution rate and routing accuracy against the prior baseline.
The Dataverse write action is a data-integrity change, so it needs more than ALM: server-side validation, duplicate detection on the returns table, error handling that surfaces failures to the user rather than silently succeeding, and a rollback path. Catalogue grounding needs permission trimming checked - users must not retrieve catalogue records they are not entitled to see. Neither ships until the regression set passes and the routing metric improves, not just the resolution rate.
2. Capacity ran out on day 22 of month one. How do you diagnose and fix consumption before buying more?
Read the worked answer
Do not buy capacity first. A 41% resolution rate means most conversations drag on, and long failed conversations are exactly what burns messages, so more capacity would fund the failure mode.
Diagnose per-conversation consumption: average messages per session, split by resolved versus escalated versus abandoned. Generative answers, knowledge retrieval calls, and action invocations each have different consumption profiles, so attribute the spend. Then forecast against 600 users and observed session frequency, and set alerts at 50, 75, and 90% of capacity rather than discovering exhaustion at day 22.
Reduce before you buy: collapse multi-turn clarification loops where the agent asks for order number, email, and postcode in three separate turns; resolve simple intents through deterministic topic branching or classic knowledge search instead of generative answers; cache high-frequency answers; and stop the agent from retrying retrieval after a failed lookup.
Then size the purchase against the corrected consumption figure plus headroom for peak periods, and put a hard quota in place so a runaway topic cannot take the agent down for everyone. Track messages per resolved conversation as the ongoing efficiency metric.
3. Leadership wants a go/no-go on expanding to a second business unit next quarter. What evidence do you require?
Read the worked answer
Require evidence that the current agent is under control, not evidence that it exists.
Stability: no unplanned outages for two consecutive months, capacity alerts firing before exhaustion, and a change history where production edits all trace to an approved pipeline deployment. Right now makers edit live, so the answer is no until that stops.
Quality: regression set passing at a defined threshold, resolution rate improving from the 41% baseline, and escalation routing accuracy improved from 72% toward 90%. Publish the trend, not a single point.
Controls: an owner accountable for the agent, a documented change approval, a DLP-reviewed connector list, and a groundedness check on the catalogue search that shows users only retrieve records they are entitled to see.
Economics: cost per resolved conversation, so expansion is justified by unit economics rather than ticket volume. If cost per resolved conversation is flat but resolution improves, expansion is defensible. If resolution is flat and consumption per session is rising, fix the design first - duplicating a leaking agent across a second business unit multiplies both the cost and the support burden.