Choosing Consistency After Adding a Second Data Center

Replicas per key: 3 in dc1 + 3 in dc2 = 6.

• Global QUORUM = floor(6/2)+1 = 4
• LOCAL_QUORUM = floor(3/2)+1 = 2 per DC
• EACH_QUORUM = 2 acknowledgements required in each DC

A read and a write are only guaranteed to overlap when W + R > RF, evaluated over the replica set actually involved.

• QUORUM/QUORUM: 4 + 4 = 8 > 6, so overlap was guaranteed - but every operation had to wait for an acknowledgement from the remote DC. That WAN round trip is the 38-45 ms p99 and the 2000 ms ReadTimeouts; local Cassandra processing did not change.
• QUORUM/ONE: 4 + 1 = 5, which is not greater than 6. No overlap is guaranteed. A dc1 replica that has not yet applied the write (its hint may still be pending) can serve the read. At CL ONE the coordinator contacts a single replica, so there is no digest comparison and no blocking read repair - the stale value is returned silently and nothing corrects it.

The latency fix is LOCAL_QUORUM on both paths: 2 + 2 = 4 > 3, so overlap is guaranteed inside dc1, and no operation waits on the WAN. Note that LOCAL_QUORUM does not stop replication to dc2; the coordinator still forwards mutations to dc2 replicas, it just does not wait for them.

Use EACH_QUORUM for writes: the coordinator must collect 2 acknowledgements in dc1 and 2 in dc2 (4 total) before returning success. That guarantees quorum durability per DC, so any single node or even a single node per DC can be lost without losing an acknowledged write.

Costs to state explicitly to the business:

1. The WAN is back in the write path. Every write now waits for the slowest of the two DC acknowledgements, so p99 write latency is bounded by inter-DC RTT plus jitter.
2. Availability coupling. EACH_QUORUM fails if any DC cannot reach its local quorum. A dc2 outage that leaves dc2 with fewer than 2 responsive replicas will reject clusters writes even though dc1 is completely healthy. With LOCAL_QUORUM in dc1 writes would keep flowing.
3. It is not the same as multi-DC consistency for reads. Reads still need a level that overlaps: LOCAL_QUORUM in dc1 (2 + 2 > 3) is fine for dc1 clients; dc2 clients reading their own LOCAL_QUORUM see everything committed at EACH_QUORUM.

A cheaper alternative that often satisfies the real requirement: LOCAL_QUORUM in dc1 (fast, always available locally) and accept that dc2 lags by the WAN latency, which is milliseconds in steady state. Reserve EACH_QUORUM for the small subset of writes where cross-DC durability at acknowledgement time is a genuine regulatory requirement.

Rule: every node must see a repair covering each of its ranges at least once per gc_grace_seconds (default 864000 s = 10 days). Eight months without repair means tombstones written in that window may already have been purged on some replicas while other replicas still hold the deleted cells.

Concrete plan:

• Run repair per DC so streams stay inside the DC and do not saturate the WAN: nodetool repair -dc dc1 on the dc1 nodes, and separately nodetool repair -dc dc2 on dc2. If you prefer per-node invocation, nodetool repair -pr on every node, and verify that every node was covered.
• In 4.x the default is a full repair; incremental repair is opt-in (--incremental). For a keyspace this far behind, do a full repair first.
• Track completion (repair history / nodetool repair_admin) and alert when any node exceeds ~7 days since its last successful repair, leaving headroom under gc_grace.
• Only consider lowering gc_grace_seconds (say to 3 days) after the repair loop is proven to run at least that often; lowering it without repair accelerates resurrection.

Why it matters: a delete writes a tombstone. Cassandra is allowed to purge that tombstone after gc_grace_seconds on a replica that has compacted it. If another replica never received the tombstone (node down past max_hint_window, or a dropped mutation), that replica still holds the live cell. Once the tombstone is purged elsewhere, the stale cell is the only surviving version and the deleted row reappears - and it will keep reappearing on every read repair and repair until it is explicitly deleted again.