Migrating 38 Million Chunks to a New Embedding Model Without Downtime
A SaaS company serves an in-app help assistant from a Databricks Vector Search Delta Sync index: 38 million chunks, 1024-dimensional embeddings from a general-purpose English model, hybrid search enabled, and a tenant_id filter on every query. Peak load is 60 QPS with a p95 SLO of 400 ms end to end. Thirty percent of queries are Spanish and quality there is poor, so the team wants to adopt a new multilingual embedding model that outputs 1536 dimensions. Chunk text lives in the source Delta table alongside the embedding column, and the index auto-syncs from that table. The application reads the index name from a config value that requires a redeploy to change. There is no staging index. Product wants cutover within three weeks, and finance has approved only a 20% increase in Vector Search spend during the transition.
1. Produce the migration plan: how do you get 38M chunks re-embedded, compare old against new, and cut over without downtime inside the budget?
Read the worked answer
Layout. Add an embedding_v2 column to the source table rather than overwriting embedding_v1. Create a second endpoint and a Delta Sync index over the v2 column in a different schema. Both indexes can then be queried in parallel, which is what the comparison requires.
Backfill. A checkpointed Spark job embeds chunks in batches through the model serving endpoint, writing v2 embeddings back with an embedding_model_version column. Delta Sync picks up changes automatically. Budget-check the embedding endpoint first: token throughput on the serving endpoint, not Spark, is the rate limit, so estimate batch size and elapsed time before promising three weeks, and use provisioned throughput if pay-per-token limits would stretch the backfill past the window.
Evaluate before cutover. Build a frozen query set stratified by language and tenant: several hundred queries with labeled relevant chunks. Compare v1 and v2 on recall@10, MRR@10, nDCG@10, and end-to-end answer accuracy. Require no regression on English and a defined improvement on Spanish. Then shadow: mirror a small slice of live production queries to the v2 index, log both result sets, and compare without serving v2 to users.
Cutover. Move the index name into a runtime lookup (a Delta config table or feature flag) so switching does not need a redeploy. Flip 5% to 25% to 100%, watching p95 and thumbs-down. Keep v1 warm for two weeks, then decommission.
Budget. Vector Search bills primarily for endpoint capacity, not per index, so the incremental cost is the second endpoint for roughly three to four weeks - inside a 20% transitional allowance if the new endpoint starts small and scales only for shadow traffic.
2. Beyond the dimension change, what else must be revisited, and what are the failure modes if it is not?
Read the worked answer
Dimension is the obvious break; the silent regressions are the dangerous ones.
Asymmetric instruction prefixes. Many embedding models expect a query-side instruction prefix distinct from the document-side text. If the old model wanted one and the new one does not - or wants a different one - recall drops with no error anywhere. Verify the model card and test both conventions on the frozen set.
Normalization and metric. Cosine similarity depends on L2-normalized vectors. Confirm the new embeddings are normalized the same way, or switch the distance metric explicitly. A mismatch produces plausible-looking but wrong rankings.
Truncation. Max sequence length and tokenizer differ between models. Chunks that fit the old model may be silently truncated by the new one, cutting off the answer-bearing sentence. Re-measure chunk length distribution in tokens under the new tokenizer.
Chunk size. Do not assume the new model's longer window means bigger chunks are better. Larger chunks dilute the embedding and hurt precision. Re-tune 512 vs 1024 against recall, not against the model card.
Hybrid weighting. The keyword arm is unchanged, but fusion weights are calibrated to the old semantic score distribution. Re-tune on the frozen set or the hybrid will quietly favor one arm.
Filters. tenant_id selectivity varies; highly selective filters can change latency behavior on pre-filtered search. Re-run the latency test with the smallest and largest tenants, not an average tenant.
3. How would you reason about capacity and cost while both indexes run, and what would make you pause the cutover?
Read the worked answer
Model cost as endpoint capacity, not storage. Two endpoints roughly double the hourly Vector Search spend during the transition, and the new one must be sized for shadow traffic on top of backfill reads. The 20% allowance is workable only if the v2 endpoint starts at the smallest usable size, scales up for the shadow window, and the v1 endpoint scales down as traffic shifts - so plan the scaling curve rather than provisioning both at full production size for four weeks.
Separate the three cost drivers: backfill (embedding tokens through the serving endpoint), dual serving (two search endpoints), and shadow traffic (extra queries, no user benefit). The first ends; the second tapers; the third is bounded by your sampling rate. Choose the shadow sampling percentage by what statistical confidence you need on the frozen metric, not by convenience.
Pause conditions, agreed in advance:
- Spanish recall@10 improves but English recall@10 drops beyond the agreed tolerance.
- p95 for the v2 path exceeds the 400 ms SLO at 60 QPS with realistic filters.
- Shadow comparison shows result-set disagreement concentrated in one tenant or document class, which suggests a data problem rather than a model problem.
- Backfill throughput projects past the three-week window at current rate limits.
Cutover should be reversible in minutes. If rolling back requires a redeploy and a reindex, the plan is not finished; moving the index name to runtime config is the cheapest insurance in the whole project.