Hybrid search regression on a 3.2M-SKU catalogue, plus a RAG assistant on the same index

Why it regressed. Two incomparable score families were summed. BM25 scores are unbounded and corpus-dependent; ELSER sparse scores are sums of learned term weights. Per-request normalisation makes the top hit of each list comparable but not the mid-list, so whichever retriever has the longer tail dominates. More fundamentally, the semantic retriever carries no lexical signal: a token like VX-4410-B contributes almost nothing to the sparse vector, so the exact SKU may not appear in the semantic list at all and only survives if the lexical contribution outweighs a semantically similar but wrong product.

Structure. One _search with a retriever tree: an rrf retriever with two children - a standard retriever whose query is a bool combining match_phrase on a part_number.text subfield (high boost) and multi_match on description, plus a semantic retriever over the semantic_text field. Fusion happens server-side and is rank-based, so score-scale mismatch stops mattering.

Parameter reasoning. RRF sums 1/(rank_constant + rank) over lists. With the default rank_constant of 60, an exact match at rank 1 of one list scores 1/61 ≈ 0.0164, while a mediocre document at rank 3 in both lists scores 2/63 ≈ 0.0317 - so pure RRF can still bury the SKU. Fix that by (a) guaranteeing the exact SKU reaches rank 1 of the lexical child via a term query on a normalised part_number.keyword, (b) using weighted RRF to give the lexical child more influence, and (c) setting rank_window_size to at least 100 so neither child truncates before fusion. Validate against a 200-query judgement set split by query type (SKU vs prose).

Measure first. Run the current query with "profile": true and compare the two client round trips against a single server-side fusion request. Then, in order of expected effect:

1. Collapse to one request. Two sequential HTTP calls each incur their own shard fan-out and coordination cost, and the application cannot run them concurrently. The rrf retriever runs its children concurrently and fuses on the coordinating node - one round trip, one fan-out.
2. Bound the work. Lower rank_window_size to the smallest value that still preserves first-page quality (start at 50), keep size/k at 10-20, and use _source includes so you are not shipping unused fields. The semantic field's query-time inference is one call per request, not per shard, so it is not the scaling problem - but an over-long query string does add latency.
3. Fix indexing pressure. A 1 s refresh on an actively written index produces many small segments and constant merges that compete with search for CPU and I/O. Move to 30 s or 60 s and accept up to a minute of staleness for catalogue updates.
4. Check where ELSER runs. If the inference endpoint is served by the same nodes that hold data, every query competes with ingestion for CPU. A dedicated ML tier or the managed ELSER endpoint removes that contention.

If p95 still exceeds 300 ms, add a second phase - retrieve 50 with RRF, rerank only the top 20 - but only after the labelled set shows a quality gain, because a rerank endpoint adds a network hop you cannot optimise away.

Retrieval. Reuse the RRF tree rather than building a parallel path: lexical child for exact part numbers and spec codes, semantic child for paraphrased questions. Then rerank the fused top 50 with a rerank inference endpoint and keep the top 5-8 chunks. Two-stage retrieval matters here because RRF optimises for recall across two very different retriever families, while an LLM needs a small, well-ordered context.

Chunking and citations. One document per SKU is too coarse: a 4,000-word datasheet diluted into one embedding retrieves poorly and burns context. Chunk by section (description, specs, compatibility, compliance) with each chunk carrying sku, source_url and section heading in its _source, and put the chunk text in its own semantic_text field so inference is configured in one place. The prompt receives those metadata fields and is instructed to cite sku and section; the application renders citations from the returned fields, not from the model's memory.

Refusal. Apply a score floor on the reranked results. If the best chunk is below it, return "not covered by the catalogue" instead of generating. Also apply the tenant/active-SKU filter to both retrievers, so a stale or forbidden record cannot be retrieved and then cited.

Evaluation. Build a labelled set of 100-200 real questions with the correct SKU and section. Track retrieval recall@8, answer groundedness (does every claim map to a cited chunk), and refusal correctness on out-of-scope questions. Re-run on every mapping, chunking or model change - otherwise quality regressions look like model behaviour.