HR Policy Assistant: Managed Knowledge Base vs a Two-Engineer Custom Stack
An HR shared-services team is building a policy assistant for 30,000 employees in the US, UK, and Germany. The corpus is 8,500 PDFs and HTML pages in OCI Object Storage, refreshed weekly; country addenda override the global handbook. Requirements: every answer cites the source passage and its version date; UK staff must never surface German works-council documents, and vice versa; content changes must be reflected within 24 hours; the team has two engineers and no ML specialists. A systems integrator proposes a custom stack - an agent framework on OCI Compute, a self-managed vector database, and scheduled ingestion jobs. The platform team suggests OCI Generative AI Agents with a knowledge base over the Object Storage bucket. Legal has asked who can prove that a UK employee never received a German document.
1. Recommend an architecture and justify it with explicit decision criteria, including the conditions under which the custom stack would be the better choice.
Read the worked answer
Criteria that matter here: time-to-value with two engineers; citation fidelity; enforceable document-level access control; freshness within 24 hours; regional data handling; operating cost; and an exit path if requirements change.
Recommendation: start with the managed service, because ingestion, chunking, vector storage, retrieval, and citation plumbing are the parts this team cannot staff. A custom stack means owning embedding model upgrades, index operations, retry logic, and a vector database's backup and upgrade path - a poor use of two engineers whose actual value-add is HR content curation.
But the ACL requirement decides the design. If the managed knowledge base only scopes access at the knowledge-base level, per-country filtering must be expressed as separate knowledge bases (or separate buckets and endpoints) per country, with routing keyed to an attribute from the identity token. That is a legitimate architecture, not a workaround, and it makes the legal question answerable: a UK request never reaches a German index.
Choose custom if you need per-user document ACLs finer than country, specialized table extraction, on-premises processing, or connectors the managed service does not support. None of those are stated requirements today.
Do a two-week spike to confirm current managed-service capabilities around metadata filtering, citation format, and connectors rather than assuming them from a datasheet.
2. Where should country-level access control be enforced, and how would you meet the 24-hour freshness requirement without weakening it?
Read the worked answer
Enforce at retrieval, twice.
- Partition the data physically: separate prefixes/buckets for US, UK, and DE, with ingestion principals whose IAM policies only allow reading their own prefix. A UK knowledge base physically cannot contain German documents.
- Filter by metadata at query time: country, doc_type, works_council flag, effective_date. Physical partitioning is the guarantee; metadata filtering handles the finer cases (works-council documents inside the UK bucket, superseded addenda).
Never rely on the prompt to exclude documents. A retrieved German passage is already in the context window and can leak through paraphrase even if instructed otherwise.
Freshness. Drive ingestion from Object Storage object events plus a nightly reconciliation job, and record index lag as a monitored metric rather than assuming the sync worked. Publish the document version date from object metadata into the citation so a stale answer is visible to the user, and expire superseded addenda by effective_date at query time so a slow reindex cannot resurface an old policy.
Test it as a security control, not a feature. Automated cases: authenticate as a UK employee, ask a question whose only answer is in a German works-council document, and assert zero citations plus a refusal. Run this in CI on every corpus refresh.
3. What must be true before go-live on citations and answer quality, and how would you structure the evaluation given the team's staffing?
Read the worked answer
Go-live gates.
- A golden set of roughly 300 questions per country with the expected source document and version. Citation precision at or above 95%, meaning the cited passage genuinely supports the claim.
- Groundedness checks: every factual sentence traceable to a retrieved passage; refusal when nothing relevant is retrieved, rather than a plausible guess.
- Supersession cases: a question where an old and new addendum disagree must cite the current one.
- Cross-country leakage cases from the security test suite, treated as release blockers.
Structure for two engineers. Do not build a bespoke eval platform. Maintain the golden set as versioned CSV in the repo, run it against the endpoint on every corpus refresh, and publish three numbers: citation precision, refusal correctness, and leakage count. Sample 30 real conversations weekly for human review and feed failures back into the golden set.
Citation mechanics. Require the answer schema to carry document id, version date, and passage id, and validate that each cited id was actually in the retrieved set - this catches fabricated citations cheaply. Surface the version date in the user-facing answer so an employee can notice staleness.
Rollback. Keep the previous index or knowledge-base version addressable so a bad refresh can be reverted in minutes.