Clinical LLM Inference: Concrete AI, Data Residency, and Budget
A Swiss hospital wants to deploy a 7-billion-parameter open-weight LLM to summarize clinical notes for 400 clinicians. Expected load is 50,000 requests per day, with p95 latency under 3 seconds for interactive summaries and overnight batch jobs for discharge letters. Notes contain protected health information and must remain in Switzerland; the hospital's regulator requires audit logs, encryption at rest and in transit, least-privilege access, and no model-provider access to prompts. The team evaluates two options in CH-GVA-2: self-managed GPU instances running vLLM behind a private network, or Exoscale Concrete AI as a managed inference service. The monthly budget is CHF 4,000, including storage and egress. The hospital has one platform engineer and no dedicated MLOps team. They also need to prove that prompts and outputs are not retained by the provider and that backups stay in Switzerland.
1. Which deployment option would you choose, and what evidence would you require before signing?
Read the worked answer
Choose managed Concrete AI only if the contract and technical configuration prove no prompt or output retention, provider access limited to break-glass with audit, and processing exclusively in CH-GVA-2. With one platform engineer and no MLOps team, self-managed vLLM is operationally risky: GPU driver updates, model serving, autoscaling, and security patching all become the hospital's problem. Managed inference is the better default if the residency and no-retention conditions are contractual, not just marketing claims.
Require a data processing agreement that names the Swiss zone, a written no-training and no-retention commitment, audit-log export to the hospital's own bucket, and confirmation that backups and logs do not leave Switzerland. Ask for the actual subprocessors and support-access model. If Concrete AI cannot provide those, fall back to a self-managed GPU instance on a private network with no public IP. In that case, budget for a second engineer or a managed operations contract, because the one-engineer constraint then becomes the main risk. Do not choose self-managed solely to avoid a procurement conversation.
2. How do you design data residency and access control for PHI?
Read the worked answer
Pin all compute, storage, and backups to CH-GVA-2. Put the inference endpoint behind a private network and a reverse proxy with mutual TLS; do not assign a public IP to the GPU host or model server. Use Exoscale IAM roles with least privilege: clinicians authenticate through the hospital identity provider to the application, and the application gets a narrowly scoped token to call inference. No clinician gets direct API keys to the model. Rotate service credentials and log administrative actions.
Encrypt data at rest on block storage and in the Object Storage bucket used for logs and backups; enforce TLS in transit. Do not log full prompts by default. Log request metadata, token counts, model version, and a pseudonymous clinician ID for audit. If debugging requires prompt capture, require break-glass approval and redaction. Apply pre-inference redaction or tokenization for direct identifiers where clinically acceptable. Store backups only in Switzerland and test that restore does not copy data to another region. Review the provider's support-access logs to prove no engineer read PHI without approval.
3. What capacity and cost plan meets the latency and budget constraints?
Read the worked answer
Estimate average load first: 50,000 requests per day is roughly 0.6 requests per second average, with a peak perhaps 5–10 times higher. A 7B model in FP16 needs about 14 GB for weights plus KV cache, so one 24–48 GB GPU with continuous batching can plausibly meet a 3-second p95 if prompts are not extremely long. Do not size for peak by buying two GPUs permanently; instead use one always-on GPU for interactive traffic and queue overnight batch jobs on the same GPU during off-peak hours.
Model the CHF 4,000 budget against the current Exoscale GPU SKU, storage for model weights, and egress. If one GPU plus storage exceeds the budget, reduce scope: cap output tokens, cache repeated summaries, and batch discharge letters. Autoscaling a second GPU during the day may be cheaper but adds cold-start and model-load time. Measure p95 with production-like prompt lengths before committing. The judgement is to separate interactive and batch queues, protect the interactive p95, and avoid over-provisioning GPUs for a load that is mostly batch.