GPU Serving Cost Reduction Under a Latency SLA
A fraud-detection model is deployed in Cloudera Machine Learning on CDP Public Cloud. It serves 1,800 requests/s at peak with a 250 ms p99 latency SLA. The workspace has three GPU nodes, each with one NVIDIA A10G, and GPU utilization averages 15%. Finance has asked for a 30% reduction in serving cost. A CPU-only test of the same model reached p99 480 ms at 400 requests/s before saturating. The model is retrained weekly, and the risk team wants to add a nightly backfill scoring 90 days of transactions. Current deployments use three replicas pinned to GPU nodes with no autoscaling, and request batching is disabled. The team proposes moving all real-time traffic to CPU and using the GPUs only for training.
1. Evaluate the proposal to move all real-time traffic to CPU.
Read the worked answer
The proposal fails the arithmetic before it reaches cost. Peak demand is 1,800 requests/s; the CPU test saturated at 400 requests/s with p99 already at 480 ms, nearly double the 250 ms SLA. Covering peak would need at least 4.5 replicas just to match the saturation point, and five replicas would leave zero headroom for variance, retries, or a node loss. Any spike would push latency further past SLA.
The cost comparison is also inverted. Three GPU replicas at 15% utilization are inefficient, but five or more CPU replicas at saturation are not cheaper per request once you include the extra pods, memory, and the operational risk of running with no burst capacity. Moving all traffic also discards the GPU's headroom exactly when batching could use it.
The valid conclusion is narrower: CPU is suitable for latency-tolerant work such as the nightly 90-day backfill, shadow scoring, and canary evaluation, but not for the 250 ms real-time path. The real inefficiency is the unbatched, always-on GPU deployment, not the GPU itself. Fix utilization and autoscaling first, then reassess whether some low-priority real-time traffic can be split off.
2. Design a serving architecture that meets the SLA and plausibly cuts serving cost by 30%.
Read the worked answer
Start from the utilization signal: 15% average GPU use with batching disabled means the accelerator is waiting on individual requests, not compute-bound. Enable request batching with a short maximum wait-for example max batch size 8 and 15 ms timeout-so p99 stays within 250 ms while throughput per replica rises. Measure the new requests/s per replica rather than assuming it.
Then reduce fixed capacity. Run two GPU replicas as the steady-state floor and add autoscaling from 2 to 4 based on queue depth or p99, not CPU. Two batched replicas should cover peak; the third replica becomes an on-demand burst rather than an always-on cost. Keep a minimum of two for availability during a node or replica failure.
Move the backfill off the serving pool entirely. Implement 90-day scoring as a CML job on CPU workers scheduled after peak, reading from the lakehouse and writing scores idempotently. This removes a large batch load from the latency-sensitive path and uses cheaper capacity.
Use the remaining GPU capacity for a 5% shadow or canary deployment rather than idle replicas. Cost reduction then comes from fewer always-on GPU replicas plus batch work on CPU; the 30% target is credible only if measured after batching, because the baseline is deliberately underutilized.
3. What evidence would you require before shifting production traffic, and how would you detect that the weekly retrain has degraded serving?
Read the worked answer
Require three evidence sets.
Performance evidence: a load test at 1,800 requests/s plus 20% burst, with p99, p999, error rate, batch wait time, and GPU queue depth reported per replica. Pass criteria are p99 under 250 ms and successful completion with one replica removed. A single average-latency number is insufficient because batching trades tail latency for throughput.
Model evidence: compare score distributions between the current and candidate deployments on a fixed evaluation set using PSI or KS, plus precision and recall at the decision threshold. A latency win that shifts scores is a business regression.
Operational evidence: a canary at 5% of traffic for at least 48 hours, with automatic rollback on p99, error-rate, or score-distribution thresholds. Include a test of the retrain promotion path, because the weekly refresh can change preprocessing and feature availability even when the model artefact looks valid.
To detect retrain degradation, monitor input feature drift, prediction distribution, and realised fraud outcomes with a lag, and alert on threshold crossings before full promotion. Keep the previous model version deployable so rollback is a routing change, not a retraining exercise.