Latency Above the Server, and a Counter That Drifts
A Java application uses DataStax driver 4.15 against a 9-node single-DC cluster with RF 3, reading and writing at LOCAL_QUORUM. Application-side p99 write latency is 90 ms, but every node reports server-side write p99 of about 4 ms. nodetool tpstats shows no dropped messages, and all hosts sit near 35% CPU. Two anomalies: request counts are uneven across hosts (some receive roughly 3x the traffic of others), and a counter table tracking page views is about 6% higher than a recount derived from the raw event table after a night that included read timeouts. Relevant code: the write path builds statements with new SimpleStatement("INSERT INTO events (...) VALUES (" + id + ", ...)"), a custom RetryPolicy retries every WriteTimeoutException once, and the counter update is UPDATE page_views SET views = views + 1 WHERE id = ?.
1. Explain the 90 ms p99 and the uneven per-host request counts, given that server-side latency is 4 ms.
Read the worked answer
The driver can only route a request to a replica when it knows the partition key. A SimpleStatement built by string concatenation carries no routing key, so the token-aware policy has nothing to work with and falls back to round-robin across all hosts. The request lands on an arbitrary coordinator, which then forwards the mutation to the three replicas and waits for LOCAL_QUORUM acknowledgements.
Consequences:
- Every write costs an extra network hop, and the coordinator does work for rows it does not own.
- Nodes that happen to receive more coordinator work show 3x the request counts - the skew is coordinator role, not data ownership.
- Server-side 4 ms is the local write path; the 90 ms p99 is hop latency plus per-host queueing at the coordinator, which is why CPU is only 35% - nothing is saturated, work is just being done in the wrong place.
Fix: prepare the statement once and bind values.
PreparedStatement ps = session.prepare("INSERT INTO events (id, ts, ...) VALUES (?, ?, ...)");
session.execute(ps.bind(id, ts, ...));
The driver caches the prepared statement per node, extracts the partition key from the bound values, resolves the token from its token map, and sends to a replica. If a statement must stay simple, set the routing key explicitly on the bound statement, and make sure all partition key components are supplied - a partial routing key silently disables routing without any error.
2. Why is the counter table 6% high, and what retry discipline should apply?
Read the worked answer
A WriteTimeoutException means the coordinator did not collect enough acknowledgements within the timeout - it does not mean the mutation failed. The increment may already be applied on one, two, or all three replicas. Retrying a non-idempotent statement turns one ambiguous write into two applications, and counters have no mechanism to deduplicate: views = views + 1 executed twice is simply +2. A 6% drift over a night of timeouts is exactly this pattern.
Retry discipline:
- Mark a statement idempotent only when re-execution is safe: an INSERT with fixed values, or an UPDATE that sets absolute values. Counter increments, read-modify-write without LWT, and list appends are not idempotent.
- Leave
setIdempotence(false)(the 4.x default) for those statements, and do not retry them on write timeouts. The built-in DefaultRetryPolicy already only retries write timeouts for BATCH_LOG writes; the custom policy is what broke the invariant. - On a timeout for a non-idempotent statement, log and reconcile rather than resend.
If exact counts matter, stop using counters here: write an immutable event row keyed by (page_id, event_id) so duplicate deliveries collapse, then aggregate in batch or in an analytics engine. Lightweight transactions give correctness but cost Paxos round trips on every increment - acceptable only at low update rates on a single key, which page views are not.
3. Which driver-level settings would you change, and which would you deliberately leave alone?
Read the worked answer
Change:
- Prepared statements with bound values everywhere. This is the single largest win and removes the extra hop.
- Explicit request timeout (
basic.request.timeout): set it above the observed p99 with margin (for example 2 s) rather than tuning around a default that hides the real problem. - Throttling instead of unbounded queueing:
advanced.throttler.class = ConcurrencyLimitingRequestThrottlerwithmax-concurrent-requestssized to what the cluster can absorb. If throttler queue metrics climb, lower the limit; if hosts are idle at 35% CPU, the limit is safe to raise. - Connection pool sizing: increase
advanced.connection.pool.local.sizeand checkmax-requests-per-connectionif metrics show per-host request queueing while CPUs are idle. - Fix the retry policy: retry only idempotent statements, never the counter increment.
Leave alone:
- Speculative execution stays off for writes and counters. It re-sends to another replica when a request is slow, which multiplies cluster load exactly when the cluster is least able to absorb it. It is reasonable only for idempotent, latency-critical reads, with the delay set above the observed p99.
- Consistency levels stay at LOCAL_QUORUM; nothing in this incident suggests weakening them, and lowering them would trade a latency problem for a correctness problem.
- Do not raise the retry count to hide the timeout. After the routing fix, revisit whether the timeouts persist at all.