Retail Knowledge Assistant: Managed Search Versus a Custom Retrieval Pipeline
A retail chain with 1.2 million product and operations documents in eight languages wants an internal assistant for store managers and a separate customer-facing answer bot. Roughly 30% of the corpus is scanned PDFs inherited from acquired regional brands. Two engineers own the project with a six-week deadline and no ML platform experience. Compliance requires every answer to cite its source, and store managers must only retrieve documents their role is entitled to see. The company already runs BigQuery and has an active Google Cloud project. A vendor demo of a fully custom vector-search pipeline impressed leadership, but the engineers estimate four months to build retrieval, re-ranking, ACL enforcement, and evaluation from scratch.
1. Recommend an architecture and justify it against the two alternatives the team is considering.
Read the worked answer
Recommendation: use a managed enterprise search service (Vertex AI Search) with two separate apps and data stores - one internal, one customer-facing - rather than one index with filtering. Separation removes an entire class of leakage bugs.
Why it wins here: connectors for common repositories, document-level access control honored at query time, extractive answers with source citations out of the box, and no ranking infrastructure to operate. It meets the compliance citation requirement and the small-team/six-week constraint.
Versus RAG Engine / hand-built retrieval on Vector Search: those give control over chunking, embeddings, hybrid search weights, and re-ranking - real advantages at scale or with unusual document structure - but the team would own ingestion, ACL filtering, evaluation, and on-call. Four months versus six weeks is decisive.
Lock-in hedge: put a thin internal API in front of retrieval so the application code does not depend on vendor-specific response shapes, and keep the evaluation set vendor-neutral. That preserves the option to migrate without a rewrite.
2. How should the team handle the scanned PDFs and the eight languages?
Read the worked answer
Scanned PDFs: OCR at ingestion, not at query time. Run documents through Document AI (layout-aware parsing) to produce clean text plus structure, store the extracted text alongside page references so citations point to a real page, and flag low-confidence pages into a human review queue. Bad OCR silently poisons retrieval - a 30% slice of the corpus is too large to skip. Exclude or down-rank pages that failed extraction rather than letting them match on noise.
Eight languages: decide between a single multilingual index and per-language indexes. A multilingual embedding model in one index is simpler to operate and handles mixed-language queries, but cross-language recall must be measured, not assumed. Per-language indexes with query-time language detection improve precision but multiply maintenance. Either way, build the evaluation set from questions written by native speakers in each market - translated benchmark questions systematically overstate quality. Keep SKUs, store numbers, and effective dates as structured metadata filters so retrieval is not asked to do exact-match work it is bad at.
3. What evidence would justify migrating to the custom pipeline later, and how would you gather it?
Read the worked answer
Migrate on measured deficits, not on demo impressions. Concrete triggers:
- Relevance ceiling: retrieval recall@10 on a labelled question set plateaus below target even after chunking and query-rewrite improvements, and the gaps cluster in document types the managed product chunks poorly (long tables, specification sheets).
- Control gaps: a required filtering semantic the product cannot express - for example dynamic row-level entitlements that change intraday.
- Cost at scale: query volume grows enough that per-query pricing exceeds the fully loaded cost of self-managed retrieval plus engineering time.
- Regulatory: a requirement for customer-managed keys or self-hosted inference that the managed tier does not satisfy.
How to gather it: maintain a labelled set of 300-500 real questions with known correct source documents; track recall@k, answer groundedness, deflection rate, and cost per resolved question monthly. Also track the operational burden honestly - if the two engineers spend a third of their time on retrieval quality issues, that is a signal too. Revisit quarterly; a decision made once and never revisited is how teams end up four months behind.