Vector memory, quantisation and filtered recall on a 60M-chunk similarity index

Arithmetic. 60,000,000 x 1,024 dims x 4 bytes = 245.8 GB of raw float32 vector data, before HNSW link structures and per-shard overheads, replicated across 30 shards on three 64 GB nodes. That does not fit healthily, and the restarts are the symptom: vectors live in native memory outside the JVM heap and compete with the filesystem cache the rest of the index needs.

Options, ranked by risk.

• int8_hnsw: one byte per dimension, about 61 GB, a 4x reduction, typically within a point or two of float32 recall when rescoring against full-precision vectors with rescore_vector.oversample at 2.0-3.0.
• bbq_hnsw: one bit per dimension, roughly 128 bytes per vector, about 7.7 GB and up to 32x. Larger recall loss, so a higher oversample and real evaluation are mandatory.
• Shard consolidation helps per-query cost (each shard carries its own graph, so 30 shards means 30 traversals) but requires a full reindex and does not fix memory on its own.

Prove it. This is a reindex anyway, so reindex into a new index with the quantised mapping and run the same 500 labelled queries against both. Accept the setting that holds filtered and unfiltered recall at or above the 0.94 baseline while meeting p99. Keep heap at or below about half of RAM and under the compressed-oops ceiling so page cache and native vector memory have room - raising heap is not a fix for vector memory, it makes it worse by shrinking the page cache.

Diagnosis. The drop appears only with a filter, so it is a search-parameter problem rather than an embedding problem. In a knn clause the filter restricts which vectors may be returned, while num_candidates is the per-shard exploration budget. With num_candidates: 100 the search explores roughly 100 near neighbours per shard and then discards those outside the tenant; if the tenant's vectors are not the nearest neighbours of the query, most of those 100 are thrown away and the returned 10 come from an effectively thin candidate set. The largest tenant is 9M of 60M globally (15%), but the per-shard distribution is probably uneven, so some shards contribute almost nothing eligible.

Fixes, cheapest first.

1. Raise num_candidates for filtered queries - start at 500-1000 - and measure recall and latency on the labelled set. num_candidates must be at least k, and this is the standard recall/latency dial.
2. Make the filter cheap and exact: filter on a keyword tenant_id, not an analysed text field, and check with profile whether the planner is doing a filtered graph search or falling back to an exact scan (which improves recall at a latency cost).
3. Change the layout: index large tenants in dedicated indices, or use custom routing on tenant_id so a query touches a subset of shards instead of all 30. This is the only option that improves recall and latency, because it cuts both candidate dilution and coordination overhead.

Do not try to fix this by increasing k alone - you will return more results of the same quality and pay for them.

Three independent contributors: refresh churn, merge pressure, and per-query work.

1. Refresh. A 1 s refresh on a growing 60M-document index produces many small segments; every segment is searched and carries its own HNSW graph. Set refresh_interval to 60 s, or to -1 during the six-hour bulk load followed by a single explicit refresh. Accept up to a minute of staleness - similarity search does not need near-real-time visibility. This is usually the single largest p99 improvement.
2. Merges. After the bulk job, and only while the data is read-only, _forcemerge to one segment per shard off-peak. Fewer segments means fewer graphs to traverse and less per-segment overhead. Never forcemerge an index still receiving writes.
3. Bulk throughput. Six hours is slow. Check bulk request size (start 5-15 MB), client concurrency (a small number of workers per data node, not dozens), and whether ingest pipelines do per-document work that could be replaced by mapping constructs. Faster ingestion also means fewer concurrent segments.
4. Query side. Fetch only the fields the UI renders via _source includes, keep k and size at 10 rather than over-fetching, and avoid re-issuing the identical query for pagination.

Sequencing. Refresh change and the reindex into the quantised mapping go in one window, since you must reindex anyway. Forcemerge follows after ingest finishes. Re-measure p99 and the labelled recall set before and after; only then revisit node count, which the budget has already ruled out. Combining forcemerge with the reindex would put two heavy I/O operations on nodes that are already restarting.