p99 of 1.2 seconds against a 150 ms contract

Diagnosis. The endpoint is not CPU-bound; it is waiting on a saturated shared database. The arithmetic is blunt: 40 requests per second × 4 enrichment queries = 160 queries per second against a 20-slot pool with multi-second queue waits. Every replica multiplies that load, so scaling out from three to six replicas roughly doubled the pressure on Impala while adding almost nothing on the compute side. The system has a shared bottleneck downstream, and horizontal scaling of the caller makes it worse - a classic case of scaling the wrong tier.

Capacity check for the CPU tier. At 30 ms of CPU per request and a 60% target utilisation, one 4 vCPU replica handles roughly 4 / 0.030 × 0.6 ≈ 80 requests per second. Peak demand of 40 rps fits comfortably in one replica. Compute was never the constraint.

Remediation, in priority order

1. Get the database out of the request path. Precompute the enrichment features into a wide, refreshed table keyed by account, or into a low-latency store. If live lookup is unavoidable, collapse four queries into one batched lookup with an IN list.
2. Pool and bound. Use a connection pool sized per replica, set hard query timeouts, and add a circuit breaker. Kerberos re-authentication per call is itself tens of milliseconds.
3. Cache. A bounded in-process TTL cache for account features absorbs repeated accounts within a burst.
4. Fail gracefully. On enrichment timeout, score with a documented default feature vector and flag the response, rather than returning a 5xx.
5. Then revisit replica count, and only after the downstream load is fixed.
6. Set the budget explicitly: 150 ms = ~5 ms parsing, ~40 ms feature fetch, ~35 ms scoring, ~5 ms serialisation, remainder as headroom.

Build v2 as a separate, immutable model build in the same project, pointing at the new artifact and, if dependencies changed, at a new pinned runtime image. Do not mutate the running build; that removes the ability to roll back.

Pre-flight, off the traffic path

• Replay a fixed sample of production-shaped payloads against v2 in a test endpoint and compare latency distribution and outputs to v1. Large per-record divergence is a signal to investigate before any user sees it.
• Confirm the new build's resource profile matches expectations and that its dependency image has been pulled to the nodes.

Canary

1. Send 5% of traffic to v2. If the platform does not support traffic splitting natively, use a routing layer or a small shadow job that scores a copy of live payloads and records predictions without serving them.
2. Compare over a full daily cycle, not ten minutes: p50/p99 latency, 5xx rate, throughput against the contract, and the distribution of predicted scores (PSI between v1 and v2 outputs, plus the churn-positive rate).
3. Ramp 5% → 25% → 100% with a defined abort criterion for each step, for example p99 above 150 ms or 5xx above 1%.

Rollback

• Because builds are immutable, reverting is a routing change back to the v1 build ID, measured in seconds. Verify the rollback path works during the canary, not during an incident.
• Keep the v1 build and its image available until v2 has run cleanly through at least one full business cycle.

After the switch

• Record the model version in every response log so any anomalous prediction can be attributed to a build.

Several problems, and they compound each other.

Exposure. A 300-feature churn payload likely includes account attributes, transaction aggregates, and possibly contact or identifier fields. Writing raw payloads to a log table creates a secondary copy of personal data outside the controls that govern the source systems. Anyone with read access to the log table can reconstruct customer-level information that Ranger is carefully protecting elsewhere.

Retention mismatch. Thirty days was probably chosen because it is a default, not because anyone decided it. If a model decision must be explainable for a longer period, the logs are insufficient; if they are not needed, they are being kept for no reason. Both directions are wrong, and a defensible answer requires stating the purpose first.

Attribution gap. If the log does not record which model build served each request, the data cannot answer the question it exists to answer.

What to change

1. Minimise at write time. Log the account key (hashed if it is an identifier), the model version, a hash of the feature vector, the prediction, the latency, and the response status. Store the raw vector only when a sampled, approved debugging flag is set, and expire it quickly.
2. Set retention by policy, derived from the audit and dispute-resolution requirement, and enforce it technically rather than by convention.
3. Restrict access on the log table to the small group that needs it, and ensure the restriction is enforced by the cluster's authorisation layer, not by the application.
4. Document it. Record the purpose, retention, access list, and the masking approach so the next audit does not have to reverse-engineer the intent.