Fabrikam: Latency, Grounding, and Autonomous Refunds in a Copilot Studio Agent
Fabrikam's service desk handles 22,000 contacts a month across chat and email. Leadership approved a Copilot Studio agent that answers policy questions from 12,000 PDFs in SharePoint and can issue goodwill refunds. In a 300-ticket replay the agent answered 71% correctly, but median latency was 9 seconds when grounding on the full document library, against a contractual 4-second target for chat. The production environment's data loss prevention policy blocks the SharePoint connector, so the makers moved grounding into the default environment, where 40 other agents also run. Finance will not permit refunds above $50 without a human decision, and auditors want every refund traceable to the policy clause that justified it.
1. Diagnose the latency and accuracy results. What would you change first, and why is a larger model the wrong first move?
Read the worked answer
Separate retrieval failures from generation failures before touching the model. Replay the 300 tickets and label each failure as wrong source retrieved, correct source but wrong answer, or no answer. If most misses are retrieval, a larger model makes latency worse and fixes nothing.
Latency of 9 seconds against a 12,000-document library usually points to oversized chunks, too many retrieved passages, and a long prompt. First changes: split the corpus by policy family into narrower indexes and route on intent so only one index is searched; use hybrid semantic plus keyword search with reranking; cut top-k to the smallest value that still passes the replay set; cache answers for the top repeated intents; and precompute answers for content that changes quarterly.
On accuracy, 71% is a baseline, not a verdict. Set a target per intent class, because refund policy questions matter more than holiday entitlement. Add citation-required output so a reviewer can see the source clause, and fail closed when confidence is low rather than answering plausibly. A larger model is justified only after retrieval recall on the replay set is above roughly 95% and answers still fail.
2. Design the refund action so that finance and audit will approve it. What controls are non-negotiable?
Read the worked answer
Treat the refund as a financial transaction, not a conversational feature.
Tiering: refunds up to $50 may be issued automatically, sampled at 10% for post-hoc review. Between $50 and $150 the agent assembles the case, cites the policy clause, and hands a pre-filled approval to a human in the loop, with no unilateral action. Above $150 it routes to a specialist queue. The threshold must be enforced server-side in the action, not in the prompt, because prompts can be talked around.
Traceability: every refund writes the order identifier, amount, citing clause, agent version, and conversation identifier to Dataverse. Audit needs the citation to resolve to a real document version, so index the policy library with versioning and store the version reference with the transaction.
Safety: idempotency keys so a retried action cannot double-pay; hard amount caps and allow-listed refund reasons in the action contract; a kill switch that disables the action without redeploying the agent; and adversarial tests in the regression set, including instructions embedded in pasted customer text. Log refusals too, since auditors will ask what the agent declined.
3. The makers relocated grounding to the default environment to bypass the DLP block. How do you resolve this without stalling the project?
Read the worked answer
Name the trade-off plainly: the workaround moved a production workload with financial actions into an environment with 40 other agents, no isolation, no ALM, and no connector governance. That is a larger risk than the original block, and it must be unwound even though it works today.
Short term: restrict the default environment to sandbox experimentation and attach a security group so new makers stop landing there. Freeze changes to the agent in that environment and document the current state as a known risk with an owner and a date.
To unblock properly, submit the SharePoint connector for approval with a clear data classification: HR and policy documents, internal tier, no customer PII. Propose either a DLP zone exception scoped to a dedicated production environment, or an approved alternative - copy the corpus into Dataverse or an Azure AI Search index fed by a governed pipeline, so the agent never talks to SharePoint directly.
Evaluate both on lead time and auditability, not convenience. If the connector exception takes longer than four weeks, the index route is likely faster and gives better control over document versioning, which the refund citation requirement needs anyway.