Deployment, MLOps, and Production NLP Systems
Deployment, MLOps, and Production NLP Systems tests the gap between a notebook model and a service that stays correct under load, drift, and cost pressure.
Serving patterns
- Batch, online (REST/gRPC), and streaming serving carry different latency contracts; quote p50, p95, and p99, never a mean alone.
- Optimizations: dynamic batching, distillation, int8 quantization, pruning, response caching, autoscaling, and GPU/CPU right-sizing.
Lifecycle
- Version the model and its tokenizer together; a mismatch silently garbles inputs.
- Containerize; use shadow or canary deployment, A/B tests, and a tested rollback path.
- Maintain a regression suite of golden inputs with expected outputs, and define explicit retraining triggers.
Monitoring
Latency, error rate, throughput, and cost per 1k tokens. Distinguish data drift (input distribution shifts), concept drift (input-to-label relation shifts), and prediction drift. Log PII-scrubbed inputs only.
Worked example
A 200 ms p95 end-to-end budget. Tokenization takes 10 ms, network 30 ms, post-processing 15 ms, leaving 145 ms. A model at 120 ms p95 appears fine but leaves only 25 ms for retries and burst queueing. Choosing a distilled model at 60 ms preserves headroom and keeps p95 stable. Reasoning: latency budgets are additive along the critical path, and queueing delay grows non-linearly near saturation, so headroom is a correctness concern.
Traps
- Monitoring uptime only; quality decays silently while all health checks stay green.
- Tokenizer version skew between training and serving.
- No fallback when the model or an upstream API fails.
- Logging raw user text that contains PII.