Bursty Image Moderation: Endpoints, Autoscaling, and Safe Rollout
SafeScreen provides image moderation for a social app. Traffic averages 1,200 images per second but spikes to 9,000 images per second within 90 seconds during live events. The SLA is p99 250 ms for a verdict, with a fallback to a second model if confidence is below 0.7. The current service runs a custom TensorFlow model on a Vertex AI Endpoint with two n1-standard-8 GPU nodes; during spikes, p99 reaches 1.8 s, 6% of requests time out, and the monthly bill is USD 120,000 because nodes stay warm 24/7. About 72% of images are near-duplicates of previously seen content. The team wants to add a new model version without downtime and must keep a human-review queue for appeals. Security requires all traffic to stay on private IPs. The next campaign is expected to double peak traffic.
1. Recommend a serving architecture that meets the latency SLA and reduces cost. Address caching, model cascade, and private networking.
Read the worked answer
The current two-node endpoint is sized for average traffic and cannot absorb a 7.5x spike in 90 seconds. Use a regional Vertex AI Endpoint with autoscaling and a private endpoint, or GKE with private nodes if finer control is needed. Put the service behind an internal load balancer and keep all traffic on private IPs.
Add a near-duplicate cache keyed by perceptual hash or embedding. If 72% of images are near-duplicates, a cache hit can return the prior verdict without a model call, reducing both latency and GPU load. Use a cascade: a fast classifier handles high-confidence cases, and only low-confidence images go to the larger fallback model. Route appeals and low-confidence cases to an asynchronous human-review queue rather than blocking the verdict path.
This architecture trades cache complexity and cascade tuning for much lower GPU cost and better burst absorption. The cache must respect deletion and appeal state so a cached verdict does not override a human decision.
2. How would you plan capacity and configure autoscaling for a 9,000 images-per-second peak that may double?
Read the worked answer
Measure single-replica throughput and p99 latency under realistic image sizes and batch settings. From that, calculate replicas needed for 9,000 and 18,000 images per second with headroom. Set a target utilization around 60–70% so replicas are not saturated before new ones become ready. Configure minimum replicas to cover baseline traffic and maximum replicas for the projected peak; pre-warm model containers and use a short but tested scale-up window.
Use load tests that ramp from 1,200 to 18,000 images per second in 90 seconds, not a steady-state test. Measure queue depth, 429s, cold-start time, and p99. If the platform cannot scale fast enough, add a queue and shed or defer low-priority traffic, or reserve capacity for the campaign.
Set alerts on latency, saturation, and replica churn. Revisit the cache hit rate and cascade ratio because improvements there reduce the required replica count more cheaply than adding GPUs.
3. Design a safe rollout for the new model version, including fallback and human review.
Read the worked answer
Register the new model in Vertex AI Model Registry and deploy it to the same endpoint without deleting the old version. Start with a shadow deployment or a small canary, such as 5% of traffic, and compare false positives, false negatives, confidence calibration, and p99 latency against the current model. Keep the old version warm for immediate rollback.
Use a circuit breaker: if p99 exceeds the SLA, error rates rise, or the new model’s disagreement rate exceeds a threshold, shift traffic back automatically. For low-confidence verdicts, route to the fallback model and then to human review. Appeals must be able to override a cached or model verdict, so store the model version and cache key with every decision.
Do not roll out to 100% until the canary has passed during at least one traffic spike. The human-review queue needs capacity planning too; a better model that creates more appeals can still degrade the end-to-end service.