The Knowledge Base That Knows Too Much and Too Little

This is a corpus and retrieval-governance problem, not a model-quality problem.

• The corpus mixes audiences. Internal runbooks were ingested alongside customer-facing articles, so retrieval has no reason to exclude customer names or hostnames.
• There is no metadata filter. Without audience, product, version, effective date, and sensitivity tags, the retriever cannot prefer current customer-safe content.
• Freshness lag is structural. Weekly article updates plus nightly indexing means up to seven days of stale answers; the retired price may still be in the index even after the article changed.
• Chunking may split context. A chunk can contain an old endpoint or a runbook excerpt without the surrounding warning or version label.
• No citation requirement or refusal path lets the model answer when retrieval confidence is low, producing the 22% uncited answers.

Retrieval is doing what it was configured to do; the configuration lacks authority and audience boundaries.

Treat grounding as a governed content pipeline.

• Separate corpora by audience. Customer-facing knowledge, internal runbooks, and support macros should be distinct sources with distinct access. Do not let the customer agent retrieve internal runbooks at all unless a sanitized derivative exists.
• Add metadata at ingestion: audience=customer, product, version, effective_date, retired_date, sensitivity, and source_owner. Retrieval should filter on audience and current version before semantic ranking.
• Chunk by section, not by page. Keep endpoint, deprecation warning, price, and effective date in the same chunk where possible.
• Set a freshness SLA. If articles change weekly, index changes on publish or at least daily; add a stale-content flag and exclude retired items.
• Require citation and support refusal. If no qualifying chunk is retrieved, the agent should say it cannot confirm and offer a human path.
• Evaluate retrieval separately from generation with precision, citation coverage, and stale-answer rate.

This makes 'current and customer-safe' an enforced property rather than a hope.

'Any product question' is too broad for a grounded support agent without a defined supported scope. The 22% uncited rate is evidence that the agent is answering beyond its evidence.

A practical approach:

1. Define supported intents by product, version, and customer segment - for example, current API authentication and pricing for supported plans.
2. Make grounding a precondition for factual claims. If retrieval yields no authoritative, current, customer-safe chunk, the agent should refuse or escalate rather than generate.
3. Use confidence and coverage thresholds to route low-evidence questions to a human or a backlog for content creation.
4. Track uncited-answer rate as a defect metric, not a style issue. Pair it with stale-answer rate, retrieval precision, and escalation reasons.

Expanding scope should follow content coverage, not precede it. The agent can handle any question by triaging it; it should only answer questions it can ground.