Deciding Whether a Model Is Ready for Production
A team has fine-tuned a language model for customer support. On their held-out test set, accuracy is 91%, up from 84% for the previous model. They want to deploy immediately. The deployment target is a GPU with 24 GB of memory, and the model in its current precision requires 30 GB. The team proposes quantizing to 8-bit to fit, which they estimate will cost about 2 points of accuracy. Latency testing shows the current model answers in 1.8 seconds at p50 and 4.5 seconds at p95, while the product requirement is under 1 second at p95. The team argues that accuracy gains justify shipping now and optimizing later. The release manager asks for a go/no-go recommendation.
1. What are the blocking issues, and how should the release manager weigh them against the accuracy gain?
Read the worked answer
There are two hard blockers and one judgement call.
Blocker 1: memory. The model needs 30 GB and the target has 24 GB. It cannot run as-is. Quantization to 8-bit is a plausible fix, but it is an unvalidated change. The estimated 2-point accuracy loss is a guess until measured. If the quantized model drops to 89%, the gain over the previous 84% model shrinks to 5 points, which may or may not justify the release. The release manager should require an actual evaluation of the quantized model on the same held-out set before approving.
Blocker 2: latency. The requirement is under 1 second at p95, and the current model is at 4.5 seconds at p95. That is a 4.5x gap, not a tuning detail. Shipping now would violate the product requirement. Quantization may help latency, but it will not close a 4.5x gap on its own; the team also needs batching, caching, or a smaller model.
Judgement call: accuracy versus readiness. A 7-point accuracy gain is meaningful, but accuracy on a test set does not guarantee quality in production. The team should also check for regressions on edge cases and safety behavior.
Recommendation: no-go for immediate release. Conditionally approve a staged rollout once the quantized model is measured on the held-out set and latency is brought under the p95 requirement through a combination of quantization, batching, and caching. Set explicit gates rather than a date.
2. Design the evaluation and rollout plan that would let the team ship safely.
Read the worked answer
The plan should have measurable gates and a staged rollout.
Evaluation gates:
Quantized model accuracy on the held-out set, reported with confidence intervals and compared against both the 84% baseline and the 91% full-precision model. Set a minimum acceptable accuracy before running the test, not after.
Latency at p50 and p95 under realistic load, including bursts. The gate is p95 under 1 second with headroom.
Memory headroom on the 24 GB target after quantization, including activation memory at peak batch size.
Quality checks on edge cases, adversarial inputs, and any regulated content categories.
Optimization steps before the latency gate: apply quantization, add request batching sized to memory headroom, add caching for repeated queries, and consider a distilled or smaller model if the gap remains.
Rollout: start with a small canary percentage of traffic, monitor accuracy proxies, latency percentiles, error rates, and user feedback, and expand only if metrics hold. Keep the previous model available for instant rollback.
Reporting: present the gates and results to the release manager in a single table so the go/no-go decision is evidence-based. The team's original argument, ship now and optimize later, is only acceptable if the latency requirement is relaxed by the product owner in writing; otherwise the blockers stand.