Vector Search and Embedding Pipelines
This domain tests the operational side of embeddings and Vector Search: model choice, embedding jobs, index types, synchronization, filtering, and lifecycle changes. The exam rewards designs that keep vectors reproducible and indexes fresh.
Embedding pipeline rules
- The embedding model fixes vector dimension and semantic behavior. Changing the model requires re-embedding the corpus and building a compatible index; never mix model versions in one index.
- Batch-embed with a Databricks-hosted embedding endpoint or Foundation Model APIs, and store chunk_id, model name, model version, dimension, and ingestion time with each vector.
- Keep preprocessing identical between ingestion and query time. Cosine similarity is common; if you normalize vectors, do so consistently.
- Track embedding cost and throughput separately from chat-model cost.
Vector Search operations
- A Delta Sync index syncs from a Delta source table and can use managed embeddings. Choose continuous sync for low-latency freshness or triggered sync to control cost.
- Direct Vector Access lets you write vectors and metadata yourself, useful for custom embedding models or external pipelines.
- Metadata columns enable filtered retrieval; confirm the index is ready before querying.
- Plan for updates and deletes: source-table changes must propagate, and orphaned vectors must not remain retrievable.
Worked example
A team has 2 million chunks, daily document updates, metadata filters on product and region, and moderate latency tolerance. Choose a Delta Sync index with triggered sync and a Databricks-hosted embedding endpoint, cosine similarity, and product and region as metadata columns. If the team later adopts a domain-specific embedding model with a different dimension, they must create a new embedding column, re-embed all chunks, build a new index, and switch traffic only after evaluation. Reasoning: daily updates do not justify continuous sync cost; a model change invalidates every existing vector.
Traps
- Changing embedding models without a full re-embed and reindex.
- Using vectors of the wrong dimension or mixing dimensions in one index.
- Assuming approximate nearest-neighbor search is exact.
- Forgetting that deletes and updates must propagate to the index.
- Using continuous sync for a corpus that changes once a day.