Deploying a 70B RAG Assistant on GPT-in-a-Box

The OOM is expected: a 70B model in FP16 needs roughly 140GB just for weights, which cannot fit on one 80GB H100. The single-replica timeout is a capacity and queueing problem, not just a model-loading problem. I would deploy the model with tensor parallelism across four GPUs, giving 320GB of GPU memory across the group, and use FP8 or INT8 quantization after an accuracy evaluation to reduce weight and KV-cache pressure. Two-way tensor parallelism on 2x80GB is possible for weights but leaves too little headroom for KV cache at 50 concurrent users, so four-way is the safer starting point.

I would serve through Nutanix Enterprise AI and KServe with at least two model replicas, continuous batching, and a bounded request queue. Re-enable a local model cache, but pre-populate it from scanned, versioned artifacts in Nutanix Objects through an approved internal path, not the internet. That removes the 40-second cold start without opening egress. Benchmark with a realistic prompt and output-length distribution; measure tokens per second, time to first token, and queue depth. If p95 first-token latency still exceeds two seconds, add replicas or reduce context length and retrieval payload before adding GPUs. The design should treat the endpoint as a pooled GPU resource, not one pod per GPU.

I would split the pipeline into an offline ingestion plane and an online retrieval plane. Documents land in a per-tenant Nutanix Objects bucket through an internal application, not through the inference namespace. An ingestion job chunks documents, calls a locally hosted embedding model, and upserts vectors into PostgreSQL with pgvector or the approved vector store. Each index build gets a version and a manifest in Objects. The online endpoint retrieves only from the active index version, so a failed daily rebuild does not corrupt the serving path.

For freshness, process changed documents incrementally using object timestamps or an event queue, and run a full reconciliation nightly or weekly. Because there is no outbound internet, all embedding models and tokenizers must be imported through the same controlled artifact process as the LLM, scanned, and versioned. I would keep embeddings and document chunks in separate buckets or prefixes with lifecycle rules, and record provenance so an answer can be traced to a document version. I would also test retrieval latency separately from generation latency; a 50-user pilot can miss its first-token target because the vector query or chunk fetch is slow, not because the GPU is slow.

I would treat model promotion as a controlled release, not a mutable deployment update. The new weights are scanned, hashed, and written to Objects with a version label and an approval record. A staging endpoint is created in a non-production namespace and evaluated against a fixed prompt set covering product facts, refusal behavior, and retrieval grounding. Only after quality and latency gates pass do we shift traffic.

For production, use a canary or blue-green endpoint: route a small percentage of internal users to the new version, compare p95 latency, error rate, and evaluation scores, then increase traffic. Keep the previous model artifacts and endpoint configuration intact so rollback is a traffic switch, not a rebuild. Log model version, index version, and request ID for every response, but avoid logging sensitive prompt content unless policy allows it. The audit package should include the approval, scan result, evaluation summary, deployment diff, and rollback test. This matters because a 70B model change can alter answer quality even when latency looks healthy.