An HR policy assistant citing superseded documents and filing duplicate escalations

Superseded policies are near-duplicates of current ones - a few words changed, structure identical - so they are maximally similar in embedding space. With no metadata, the retriever has no basis to prefer the current version, and top-k of 3 will happily return two old versions and one new one. Fixed 512-token chunks compound this by cutting across section boundaries, so a chunk may contain the tail of one condition and the head of another.

Redesign:

• Chunk by structure, splitting on headings and clause boundaries with modest overlap, so each chunk is a coherent rule. Typical policy sections land well under 512 tokens.
• Attach metadata at ingestion: document ID, policy family, version, effective date, superseded flag, region, and owning department.
• Filter hard at query time - superseded = false and effective date on or before today, scoped to the employee's region - enforced in the retriever, not merely requested in the prompt. A prompt instruction is a suggestion; a metadata filter is a guarantee.
• Retrieve broadly then rerank: pull 20 candidates, rerank, and pass the best 4–5 with their document IDs and effective dates into the prompt.
• Add lexical search alongside vectors, because policy questions often hinge on exact terms such as a specific leave code.

Deduplicate near-identical versions at ingestion and record the supersession relationship, so an old document is retrievable for legal audit but never for answering an employee.

The invented entitlement is a grounding failure: the model answered where the retrieved context did not support an answer, or it blended two policies. Fix both sides.

Generation side:

• Instruct the model to answer only from the provided context, quote the supporting clause, and say explicitly that it cannot find the answer when the context is insufficient. Make abstention an acceptable outcome - reviewers punish confident wrongness far more than a referral.
• Set temperature to 0 for factual policy answers.
• Add a post-generation check that every cited document ID exists in the retrieved set and was not flagged superseded; fail the answer rather than display it if the check breaks.
• Return the citation with version and effective date in the response, which is what legal is asking for.

Evaluation side:

• Build a labelled set of a few hundred questions with expected source documents, spanning all policy families and regions, including deliberately unanswerable questions to test abstention.
• Measure groundedness/faithfulness (is every claim supported by retrieved context), answer relevance, and context relevance separately, using the generative-AI quality evaluations available in watsonx.governance with an LLM judge, calibrated against a human-labelled subset. Judge scores that disagree with humans are not evidence.
• Track citation accuracy - the share of answers citing a current, correct document - as a first-class metric, since that is where the 34 percent sits.

Iterate against the same set. Ninety percent acceptance by reviewers is then a measured claim, and you can show which change moved which metric.

The tool is not idempotent and the agent retried blindly. The escalation call timed out, the agent could not tell whether the write succeeded, and its retry policy fired - twice. The ticketing API created a new case each time because nothing identified the request as already submitted.

Fixes, in order of importance:

1. Idempotency. Generate a deterministic key per escalation intent (employee, case reference, time window) and pass it to the API, which returns the existing case instead of creating another.
2. Check before write. Expose a read tool for open escalations and instruct the agent to query it before filing; require a positive absence before the write.
3. Retry discipline. Retries with backoff only for errors known to be transient, and a hard cap on side-effecting calls per conversation. A timeout is ambiguous - resolve it by reading state, not by re-writing.
4. Human confirmation. Any irreversible action gets explicit user confirmation with the parameters shown, plus a short undo window or an approval queue.
5. Least privilege. The agent authenticates with a scoped service ID that can create escalations but not close or reassign them, and every tool call is traced with its inputs and outcome.

Also tighten tool descriptions and parameter schemas - vague descriptions cause malformed calls in the first place. Then write regression tests for the timeout path, since this failure will recur whenever a dependency is slow.