Read-your-writes and connection churn in a Java order service

Why it happens: N1QL reads from GSI indexes, not from the data service. With the default scan consistency (not_bounded), the query returns whatever the index has ingested so far, and index ingestion trails the KV write by milliseconds to about a second. The write succeeded; the index simply had not caught up.

Options, cheapest to strictest:

1. KV get by key. If the page only needs the order just placed, read it with collection.get(orderId) and merge it into the query results. KV reads are served by the active vBucket and see the write immediately.
2. at_plus with the mutation token. The upsert result carries a mutation token; pass it with the query so the index waits only for that mutation. Precise and cheap, but the token must travel from the write service to the read service - Kafka headers, for example. Real plumbing, not a one-line change.
3. request_plus. The query waits for the index to catch up with all mutations known at request time. Simple and correct-looking, but it adds latency proportional to write rate and makes reads hostage to index health.

Recommendation: option 1 for the "just placed" case, since the UI already has the order ID. Reserve request_plus for the history query, and only if the product cannot tolerate a stale list. Do not apply it cluster-wide - it turns every read into a synchronization point.

Connection lifecycle: a Cluster object bootstraps, authenticates, opens sockets, and runs a config poller. Creating one per request repeats that handshake, wastes sockets, and multiplies the number of objects that must independently notice a topology change. Cluster, Bucket, Scope, and Collection are thread-safe and long-lived: create one Cluster per JVM at startup, call waitUntilReady, and inject the collection.

Why the 02:00 burst: when the node restarts, in-flight requests to it fail and the SDK receives a new cluster map to re-route with. Thirty to sixty seconds of TimeoutException suggests requests stuck on dead connections plus pods still bootstrapping new clusters instead of reusing a reconfigured one. With the default KV timeout, a slow re-route looks like an outage.

Changes:

• Singleton cluster bean with waitUntilReady(Duration.ofSeconds(10)), failing fast at startup.
• Set kvTimeout just above p99 (e.g. 500 ms) so failures surface quickly; set queryTimeout explicitly instead of inheriting the long default.
• Configure a retry strategy with capped exponential backoff and an overall deadline, plus a circuit breaker at the API edge.
• Let the SDK handle reconfiguration; do not recreate the cluster on TimeoutException.

Durability: upsert without a durability requirement is acknowledged once the active node holds the mutation in memory. If that node dies before replication, an order the API returned 200 for is gone. The stated requirement makes durability part of the contract.

Options:

• durability(DurabilityLevel.MAJORITY) - acknowledged after replication to a majority of the configured replicas. With replica count 1 that means active plus one replica. This matches "survives loss of any single node".
• MAJORITY_AND_PERSIST_TO_ACTIVE adds persistence on the active node; stronger, slower, appropriate if a node can lose power.
• Older observe/PERSIST_TO_MAJORITY polling works but the SDK 3 built-in durability level is the cleaner API.

Costs and constraints: each durable write adds a network round trip to the replica - expect a few extra milliseconds at p99, so measure. It requires replica count ≥ 1 on the bucket; on a zero-replica bucket MAJORITY cannot be satisfied.

The interaction the team has wrong: durability is about surviving failure, scan consistency is about visibility. A durable write is still not in the GSI index a millisecond later, so the history query can still miss it. Fixing durability does nothing for the stale read, and request_plus does nothing for node-loss durability. They are two independent settings that both need to be deliberate.