HR assistant citing superseded policy and timing out

Root cause: recency is not represented anywhere in the pipeline.

• Vector similarity measures topical closeness, not validity. Fourteen near-identical leave-policy chunks will all score highly; whichever wins is effectively arbitrary.
• The prompt does not ask for citations, so neither the model nor the reviewer can trace which revision produced the answer, which inflates the apparent error rate.

Fix (layered):

1. Metadata at chunk level: effective_date, superseded_date, status, doc_id. Set status=current during ingest.
2. Filter at query time, not prompt time: apply $filter: status eq 'current' so superseded chunks never reach the model. Do not ask the model to 'prefer recent documents' - it cannot see dates reliably and will not comply consistently.
3. Hybrid + semantic ranker for the remaining candidates; semantic ranking handles the phrasing mismatch in HR questions better than pure vectors.
4. Require citations in the system prompt and return doc_id/chunk_id in the response, so reviewers can verify.

Caution: keep superseded versions indexed if audit questions ('what did the 2021 policy say?') are in scope - filter them out only for current-policy answers.

Where the time goes - hypothesise before tuning.

Top-k=50 chunks at ~500 tokens each is roughly 25,000 tokens of context per request. Prompt processing at that size dominates latency; the embedding call and index lookup are milliseconds by comparison. So the biggest lever is shrinking the context, not upgrading hardware.

Ordered changes:

1. Reduce final context to 4-6 chunks. Retrieve 50 with hybrid search, let the semantic ranker cut to the top 5, and send only those. Expect the largest single improvement.
2. Set temperature to 0 (or ~0.2). The task is extraction from documents; sampling buys nothing and adds variance in length, which adds latency.
3. Cap max_tokens on the response and stream output so perceived latency drops even if total time is unchanged.
4. Chunk smaller (~300-500 tokens with overlap) so five chunks carry less irrelevant text than five large ones.
5. Cache frequent identical questions and their embeddings.

Validate: instrument retrieval time, prompt-token count, and completion time separately. Raising top-k or adding reranking without cutting the final context often makes latency worse, not better.

Retrieval is the wrong tool: the answer is not in any document.

• Leave balance is per-user, per-day state in the HR system. Indexing it is both stale and an access-control problem - a shared index would leak one employee's balance to another.
• Convert to an agent with a tool: register an HR API function (get_leave_balance(employee_id)), let the model choose it via function/tool calling, and inject employee_id server-side from the authenticated session, never from model output or user text. Authorise at the API with the caller's token.
• Constrain the agent: a clear tool description ('use for leave balance questions'), a routing rule that forbids answering balance questions from documents, and a fallback that says 'I can't retrieve that' rather than guessing.

Groundedness programme:

• Require inline citations and score groundedness on a sampled evaluation set (e.g. 200 questions monthly) rather than trusting spot checks.
• Add a refusal path when no chunk clears a relevance score threshold.
• Separate evaluation sets for 'policy' questions (retrieval) and 'personal data' questions (tool), because a single blended score hides regressions in the smaller category.