Retrieval Quality and RU Budget for a Ticket Assistant

Why the PoC failed: a UDF cannot use an index. SELECT ... WHERE udf.cosine(c.embedding, @qv) > 0.8 deserializes every document, decodes base64, and computes the distance in JavaScript for all 2.2 million chunks - a full scan with a scripting penalty, on documents whose embeddings were already inflated ~33% by base64. There is no index the engine could have used, so 2,800 RU and 9 seconds is the expected outcome, not an anomaly.

Container definition (create it with these; the vector policy is not fully mutable afterwards without recreation):

"vectorEmbeddingPolicy": { "vectorEmbeddings": [{
  "path": "/embedding", "dataType": "float32",
  "dimensions": 1536, "distanceFunction": "cosine" }]},
"indexingPolicy": {
  "includedPaths": [{"path": "/tenantId/?"},{"path": "/ticketId/?"}],
  "excludedPaths": [{"path": "/*"}],
  "vectorIndexes": [{"path": "/embedding", "type": "diskANN"}]
}

Two details that bite people: the embedding path must be excluded from the standard indexing policy (it is covered by the vector index, and a range index on a 1536-element array is pure write cost), and the query's embedding must come from the same model and dimension as ingest - a mismatch is either a hard error or silently poor results.

Query:

SELECT TOP 5 c.chunkText, VectorDistance(c.embedding, @queryVector) AS score
FROM c
WHERE c.tenantId = @tenantId
ORDER BY VectorDistance(c.embedding, @queryVector)

Because the filter is on the partition key, the vector search is scoped to a single logical partition: hundreds of megabytes for the largest tenant, tens of megabytes for the rest, which is where DiskANN is fast and cheap - single-digit to low-double-digit RU and well inside 150 ms p95. Store embeddings as native number arrays, not strings, and batch-insert them; the write cost of a float32 1536-vector is roughly 6 KB on top of the chunk, so weigh 1.5 KB chunks against larger chunks that carry fewer embeddings.

The aggregation queries are competing for the same 30,000 RU/s as retrieval, and because they are cross-tenant scans they consume large amounts of throughput and trigger 429s that the interactive path cannot absorb.

Fix: get analytics off the transactional path with the analytical store. Enable it on the container and let analysts query via Synapse Link or a Fabric mirrored database. The columnar store is separate, so aggregation consumes no RU from the operational budget and cannot throttle retrieval. Two constraints to state plainly:

1. It is not retroactive by default. The analytical store syncs from the moment it is enabled, so the two years of historical tickets are not there unless you backfill by re-writing them through a change-feed-based copy into a container that had the analytical store enabled from the start. If analysts need immediate history, plan that backfill explicitly - otherwise the first answer to them is 'from today forward', which is usually unacceptable.
2. Analytical TTL is separate from operational TTL, and the store bills for storage; set it to the retention the analysts actually need rather than the operational retention.

If enabling the analytical store is blocked, the fallback is a change-feed-maintained aggregate container (per tenant, per month, per category counts) so the dashboards read small documents. That is more work and answers only the questions you anticipated, whereas the analytical store answers ad hoc questions.

Separately, guard the retrieval path: cap the SDK's query RU with a per-request budget and handle 429s with x-ms-retry-after-ms backoff so a chat user's request degrades to a slower answer rather than a failure while analytics are running.

Decision rule: it comes down to whether vector retrieval is always scoped by a partition key you already query on, and whether you need lexical ranking.

Keep it in Cosmos DB when:

• Every retrieval query filters on tenantId (or another partition key) before the vector search, as this one does. You get one system, one consistency story, no dual writes, no re-indexing pipeline, and per-tenant isolation for free.
• Chunks per tenant stay in the hundreds of thousands - this tenant's 400,000 chunks at 1.5 KB is under a gigabyte, comfortably inside a logical partition.
• The embedding model and dimensions are stable and you can afford to recreate the container when they change.

Move to Azure AI Search when:

• You need hybrid retrieval - BM25 keyword plus vector plus semantic re-ranking. VectorDistance alone gives you semantic similarity, not relevance ranking, and support tickets are full of exact tokens (error codes, SKUs) that embeddings retrieve poorly.
• You need cross-tenant or cross-container retrieval, faceting, or filters well beyond what the index policy supports.
• The corpus outgrows a single logical partition's 20 GB limit, or you need to change embedding model/dimensions without a migration.

Hidden cost of AI Search: you now own a second index and the pipeline that feeds it. Every chunk write becomes a Cosmos DB write plus an indexing call, and the two can diverge - a ticket deleted in Cosmos DB still surfaces in search results unless your change-feed deleter is correct. You also inherit a second thing to secure, monitor, and pay for, and a staleness window that users will notice when they edit a ticket and immediately search for it.

A defensible middle: ship Cosmos DB vector search now because it meets the latency and RU targets and removes an entire failure domain, and revisit when relevance complaints (not latency complaints) start arriving - that is the signal that hybrid ranking, not a faster index, is what the product needs.