2 million nightly scores and a 150 ms storefront API from one model

Use a batch endpoint with a batch deployment, not the existing online endpoint and not a hand-rolled script loop on a compute cluster.

The sizing argument is concrete. 2 million rows in 120 minutes is about 278 rows/second sustained. The real constraint is not row count but model load and per-invocation overhead: a 900 MB ensemble takes tens of seconds to deserialize, so it must be loaded once at process start, and the scoring script must process a large mini-batch per invocation rather than one row per call.

Configure the batch deployment with:

• mini_batch_size sized so each invocation does thousands of rows (e.g. 10k-50k), amortising inference overhead.
• Autoscale on instance count (or a fixed 4-8 instances) rather than a single large node, so you can also use the max_concurrency_per_instance knob.
• Output written to a registered data asset / datastore path so downstream jobs can reference a version.

Then measure on a 1% sample first and extrapolate; do not discover the throughput at 1 a.m. If the extrapolation misses the window, raise instance count - this workload is embarrassingly parallel, so it scales nearly linearly until the storage side saturates.

Mixing the nightly job into the online endpoint would risk the p99 SLA and force 24/7 capacity for batch work.

Unknown, and unresolvable by reasoning alone - but the levers are identifiable in advance.

Arithmetic: 40 rps across 2 instances is 20 rps per instance, i.e. 50 ms of budget per request per instance if perfectly parallel. A p99 of 150 ms with 4 vCPUs means the scoring path must be extremely tight. The 900 MB model per replica fits in 14 GB, so memory is fine; the risk is CPU-bound inference plus Python request overhead, and p99 is dominated by queueing, not by mean latency. Once utilisation passes roughly 60-70%, queueing makes p99 climb sharply - so headroom matters more than average throughput.

Find out with a load test against the deployed endpoint at 1x, 2x, and 3x expected peak, measuring p50/p95/p99 per instance and watching CPU and memory. Do not extrapolate from a local benchmark.

Levers if it fails: increase instance count and add autoscale rules (CPU > 60% or p99 > 100 ms); ensure the model loads at startup, not per request; enable request batching in the scoring script if the workload allows; reduce per-request feature computation by precomputing store/SKU features. If p99 still fails, the fallback is to shrink the model - fewer trees, or export to ONNX - since inference cost scales with ensemble size.

On idle capacity, be honest about the options. Managed online endpoints do not scale to zero; the minimum instance count is at least one. So the realistic saving is going from two instances overnight to one, not to zero. Implement that with an autoscale profile on a schedule (min 1 / max N by day-part) rather than trying to engineer around the platform. Verify the minimum-instance behaviour in the docs for the specific endpoint type before promising a number to Finance - if scale-to-zero turns out to be unsupported, the alternative is accepting one instance overnight or routing overnight storefront traffic elsewhere.

On safe rollout, use the endpoint's multiple deployments:

1. Deploy the new model version as a second deployment on the same endpoint and validate it with direct (non-traffic) calls.
2. Shift 10% of traffic via the endpoint's traffic-allocation rule, and hold it there long enough to observe p99 and error rate at realistic load - 10% of 40 rps is thin for p99, so also run synthetic load against the new deployment.
3. Promote to 100% only if p99 and error rate hold; rollback is an immediate traffic shift back, no redeploy.

Prefer a smaller or quantised model only if the load test shows the current one cannot hold p99 with headroom - accuracy is the constraint, not just cost.