Latency Budget Squeeze on a Multimodal Support Assistant
You run a customer-support assistant that accepts a photo plus a short text question. Traffic is 12 requests per second at peak. The pipeline is: speech-to-text for voicemail uploads (about 30% of requests), an image-captioning model, a text LLM that composes the reply, and text-to-speech for 20% of replies. End-to-end p95 latency is 6.8 s against a 4 s target. Telemetry shows: STT p95 1.1 s, captioning p95 2.4 s, LLM p95 2.9 s, TTS p95 0.9 s, and these stages run strictly sequentially. Error rate is 0.4%, all from captioning timeouts. Product wants to keep every feature.
1. Where is the latency actually coming from, and what is the first structural change you would make?
Read the worked answer
The stages are sequential, so p95 values roughly stack: 1.1 + 2.4 + 2.9 + 0.9 ≈ 7.3 s, consistent with the observed 6.8 s. The single largest contributor is the LLM (2.9 s), but captioning (2.4 s) is the one failing.
Key insight: captioning and the LLM are not truly dependent in the common case - the text question can be embedded and routed while the image is still being described. Running captioning and text preprocessing concurrently, then joining before final composition, removes roughly min(2.4, text-prep) from the critical path. Also, STT and TTS apply to disjoint request subsets, so they never both sit on one request's path; don't sum them as if they do for a single request.
First structural change: parallelize captioning with text-side work and enforce a captioning deadline (e.g., 1.5 s) that degrades to text-only answering rather than timing out the whole request. That attacks both the p95 tail and the 0.4% error rate at once.
2. A teammate proposes switching the LLM to a smaller model to cut 1.5 s. Evaluate this against alternatives.
Read the worked answer
It works arithmetically but is the riskiest lever, because the LLM is the quality-critical stage and the change is invisible until you measure answer quality. Sequence the options by risk-adjusted payoff:
- Parallelize captioning and text prep - no quality loss, likely 1.5–2 s saved. Do this first.
- Deadline + graceful degradation on captioning - converts tail timeouts into text-only answers, fixing the error rate without touching model quality.
- Streaming TTS / streamed LLM output - improves perceived latency even if wall-clock p95 barely moves; relevant since TTS is on 20% of replies.
- Smaller LLM - only after 1–3, and only with an eval set comparing answer quality on image-grounded questions.
Also check whether captioning is over-provisioned: 2.4 s p95 for captioning suggests a large model or cold capacity. A distilled captioner may be acceptable since captions feed the LLM, not the user directly - a lower-stakes substitution than swapping the composer.
3. How would you validate the 4 s target before shipping?
Read the worked answer
Measure the right thing: p95 end-to-end per request class, not per-stage. Split traffic into (a) text-only, (b) image+text, (c) voicemail+image, since STT only affects (c) and TTS only affects the fraction of replies spoken. A blended p95 hides the worst class.
Build a replay harness at 12 rps peak with realistic image sizes and voicemail lengths, and assert: p95 < 4 s for each class, error rate < 0.1%, and captioning-deadline fallback rate reported (if it's high, users are silently getting worse answers). Add a quality gate: a labeled set of image-grounded questions scored before/after, so degradation is a deliberate, measured trade rather than an accident.