70B agent on 4×L40S: KV pre-allocation, INT4 regression and the concurrency target
Atlas is deploying a 70B-parameter agent model on-premises on 4×L40S (48 GB each) using TensorRT-LLM in Triton. The workload is tool-heavy: median prompt 6,000 tokens, 300-token outputs, 15-turn cap, p99 prompt-plus-output 6,900 tokens. The engine is configured with a 32,768-token maximum sequence length. Profiling shows only 9 concurrent sequences fit before OOM, so p95 latency at 40 users is 22 s and 31% of requests hit the queue timeout. Raising concurrency triggers OOM. An INT4 AWQ build fits in 2 GPUs, but offline evaluation shows tool-argument exactness dropping from 92% to 86% on the 300-case suite. Product needs 60 concurrent users. A 70B FP8 build passes the suite at 92%. An 8B model is also available. Budget allows one additional 4-GPU node next quarter.
1. Do the memory reasoning and choose a configuration. What is the cheapest change that gets to 60 concurrent users?
Read the worked answer
The constraint is not weight memory; it is over-provisioned KV. A 70B model in FP8 occupies roughly 70 GB, so with tensor parallelism 4 each L40S holds about 17.5 GB of a 48 GB card, leaving roughly 120 GB across the node for KV cache and activations. At 32,768 configured context, a single sequence can reserve on the order of 10 GB of KV; 120 GB divided by that is about a dozen sequences, consistent with the observed 9 after activations and fragmentation.
The p99 need is 6,900 tokens. Setting max_seq_len to 8,192 cuts per-sequence KV reservation by roughly four times, taking the node from about 9 to roughly 35–40 resident sequences - for the price of a configuration change plus a load test. No quality impact, because nothing in the observed distribution approaches 8,192 tokens; the 32,768 figure was a default, not a requirement.
Ordering: (1) right-size max_seq_len and re-measure, (2) enable paged KV cache with a memory fraction tuned to leave headroom to avoid OOM under burst, (3) if 60 users still miss, add the second node next quarter rather than accepting the INT4 quality loss now. The cheap fix should be exhausted before buying capacity or degrading accuracy.
2. How should the INT4 tool-argument regression (92% to 86%) be interpreted and handled?
Read the worked answer
Before treating 86% as a quality floor, check whether the failure is formatting or reasoning. Tool-argument exactness is often measured by string equality, which penalises "$1,200.00" versus 1200 and schema-equivalent orderings. Re-score the 300 cases with schema-aware comparison and constrained/structured decoding enabled on the INT4 build. If the gap narrows materially, the regression was a serialisation artefact; if it holds, it is a real capability loss on argument selection.
Inspect the failures by category rather than in aggregate. Arguments with many free-form fields, or with values that must be copied exactly from context, are typical INT4 casualties - that matters because payout amounts, dates and record IDs are exactly those fields.
If the gap is real, do not route the tool-calling turn to INT4. Options: keep the 70B FP8 as the planner and argument generator and use INT4 only for summarisation, classification and drafting where the eval shows parity; or use the 8B model strictly for routing and retrieval reformulation, never for argument construction. Splitting by task role is cheaper than splitting by tenant, and it confines the weaker model to steps where exactness is not load-bearing.
Whichever path, the 300-case suite becomes a release gate, with a per-field breakdown, not a single pass rate.
3. Design the rollout to production with 60 concurrent users, including what you would canary and what would make you roll back.
Read the worked answer
Phase 0, shadow: run the new configuration on mirrored traffic at low concurrency, comparing outputs against the current build on the 300-case suite and on live traffic sampled daily. Gate: no per-field regression in tool arguments, schema-violation rate under 0.5%.
Phase 1, canary at 5% of sessions for 48 hours, then 25%, holding the query distribution fixed so latency comparisons are meaningful. Gates evaluated per step: p95 end-to-end latency under target, queue-timeout rate under 1%, tool-argument exactness at or above the shadow baseline, schema-violation rate within threshold, and escalation-to-human rate not rising (a rise is the earliest signal that answer quality dropped).
Rollback triggers, automatic and pre-agreed: queue-timeout rate above 5%, OOM events on any replica, schema-violation rate above 1%, or tool-argument exactness below baseline by more than 1 point on the daily sampled set. Rollback path is the previous engine config plus pinned model artefacts, and it must be exercised at least once in staging.
Phase 2, steady state: monitor KV occupancy against the max_seq_len assumption. If prompt lengths drift upward - for example a new document type pushes p99 past 8,192 - the capacity analysis is invalid and must be rerun before scaling further.