Endpoint Latency SLO Against Deployment Sizing and Bandwidth

Bandwidth arithmetic: 10 Mbps is 1.25 MB/s. An 8 MB request takes about 6.4 seconds of transfer before the model sees a single token, and the response adds to that. At p95 of 9.2 s, roughly two-thirds of the budget is network, not inference - the 300 ms model time is noise.

Why scale-out failed: each instance carries its own bandwidth ceiling, so adding instances adds parallel capacity but does not speed up any single large request. Autoscaling also cannot trigger here: CPU is at 18%, and a policy keyed on CPU sees an idle deployment while requests queue in transfer. A single request is not divisible across instances, so horizontal scaling cannot fix a per-request size problem.

The hidden second cost: re-initializing the tokenizer per invocation and reading a 400 MB dictionary from block storage adds seconds of I/O and memory pressure per call, and it multiplies under concurrency. At 40 requests per minute, that pattern alone can saturate the block volume.

Diagnosis before changing anything: instrument request size, transfer time, tokenizer load time, and inference time separately. The fix depends on which of those dominates for the 6–9 MB cohort; guessing a bigger shape would raise cost without touching either bottleneck.

Split the paths. Small requests stay synchronous and inline. Large documents should not travel as a JSON body: the client uploads to Object Storage using a pre-signed URL and calls the endpoint with a reference, and the endpoint streams the object. Alternatively, large documents go to a batch job that scores them and writes results back, with the API returning a job handle. Either removes the megabyte payload from the latency-critical path.

Configuration changes:

• Load the tokenizer once at module scope so it is initialized per instance, not per request; bake the dictionary into the artifact or keep it on the instance's local storage rather than re-reading it.
• Raise bandwidth to a level justified by the arithmetic - for an 8 MB payload inside a 3 s budget, 10 Mbps is impossible, so size upward and re-measure.
• Right-size the shape to memory need (model plus tokenizer plus concurrency headroom), not to CPU percentage; 18% CPU means CPU is not the constraint.
• Set autoscaling on a metric that reflects queueing and latency rather than CPU alone, and validate with a load test that includes the large-payload cohort.

Client changes: raise the timeout to cover the true p99 of the chosen path, add retries only for idempotent calls, and stream or chunk uploads. A 10-second client timeout under a 14-second p99 guarantees a steady tail of failures regardless of server tuning.

Answer with arithmetic and a stated assumption, not with optimism. For a single 8 MB synchronous request, the transfer alone consumes most of a 3-second budget at any realistic bandwidth, leaving thin margin for parsing, tokenization, and inference. Meeting 3 s for the large cohort requires either a much higher bandwidth with a shaped payload or removing the large documents from the synchronous path entirely.

So the defensible commitment is: 3 s p95 for requests under, say, 100 KB, with large documents handled asynchronously and returning a result within an agreed longer window. That is a product change, not a tuning change, and it needs the client team's agreement - which is why it must be raised now rather than after the SLO is signed.

Evidence to present: a load test at the burst rate of 40 requests/minute with the real payload mix, reporting p50/p95/p99 split by size class, plus cost per instance-hour at the proposed shape and bandwidth. Include the failure mode at the 06:00 downstream deadline: what the caller does when the endpoint returns a handle instead of a score, and what happens to in-flight work during a deployment.