Peak-Hour Throttling on a Claims Summarization Workload
A claims platform on OCI uses the Generative AI service's on-demand inference with a Cohere Command R+ chat model to summarize claim packets and extract injury codes. Volume is 9,000 packets per day, averaging 6,200 input tokens and 400 output tokens. 68% of traffic lands between 09:00 and 13:00. The p95 latency target is 8 seconds; production shows 14-22 seconds and sporadic HTTP 429 responses at peak. Documents are re-summarized every day even when unchanged. Two proposals are on the table: switch to a smaller, cheaper chat model, or move to a dedicated AI cluster. Finance wants a cost comparison at current volume and at 3x volume. The team has a 200-example golden set with human-written summaries and a 96% acceptance bar. The current model scores 94%; the smaller model scored 88% in a quick test, but that test truncated each packet to 2,000 tokens.
1. What most likely causes the 429 responses and the latency spike, and what should be measured before any model or capacity change?
Read the worked answer
Start from arithmetic and telemetry, not from the model. Nine thousand packets x roughly 6,600 tokens is about 59M tokens per day; with 68% arriving in four hours, peak throughput is roughly 40M tokens over 240 minutes, near 165K tokens per minute before retries. On-demand inference is shared capacity governed by per-tenant, per-region limits expressed in requests and tokens per minute; when a burst exceeds them the service returns 429 instead of queueing indefinitely. Client retries without exponential backoff and jitter multiply the offered load and stretch p95, which is why latency and 429s appear together. Before changing anything, measure at 5-minute granularity: tokens and requests per minute, concurrency, retry counts, time-to-first-token versus total time, and finish_reason distribution. Then measure the cheapest win: unchanged documents are re-summarized daily. Hash each document and cache the summary keyed by hash plus prompt version; removing 40-60% of calls changes the capacity question entirely.
2. How should the team decide between the smaller model and a dedicated AI cluster?
Read the worked answer
Judge each proposal against the acceptance bar and the traffic shape. The smaller model scored 88% against a 96% bar, and it was tested on truncated documents, so the comparison is invalid twice: quality fails and the test did not represent production input. Re-test it on full 6,200-token packets; if it still misses the bar it is out regardless of price, because a wrong injury code has downstream cost. The dedicated AI cluster is the credible fix for the symptom: it reserves capacity on hardware you control, so p95 latency stops depending on other tenants and throughput stops depending on shared quotas. Its economics are fixed per unit-hour, so it wins only when utilization is sustained. Compute the break-even: cluster cost per hour divided by on-demand cost per million tokens gives the token volume at which dedication is cheaper. At 59M tokens per day with 68% in a four-hour window you are likely above it, but run the caching change first, because caching may drop you below break-even.
3. What changes at 3x volume, and how should the architecture be adjusted?
Read the worked answer
At 3x volume, roughly 178M tokens per day and about 500K tokens per minute at peak, on-demand becomes a quota negotiation and a tail-latency gamble. The dedicated cluster's cost scales in unit steps rather than linearly, so headroom bought for peak sits idle off-peak. Two design moves reduce the peak instead of paying for it: queue non-urgent packets into an asynchronous backlog processed overnight, reserving synchronous calls for adjuster-facing requests; and cache and batch aggressively. Then size the cluster for the synchronous peak only and use on-demand as overflow rather than as the primary path, so a 429 triggers a fallback model or a queue rather than a blind retry. Confirm that a hosting cluster serves one model at a time, so summary and extraction workloads must either share one prompt or be planned as separate capacity. Finally, re-run the 200-example golden set after every change; a latency fix that degrades extraction accuracy is not a fix.