Consolidating Three Model Formats onto One Serving Layer
A retail platform serves three models: a demand forecast in scikit-learn, a product classifier in TensorFlow, and a reranker in PyTorch. The platform team wants one serving layer on Kubernetes. Peak traffic is 200 requests/second, p95 latency SLO is 120 ms; baseline is 20 requests/second. The SLO is measured at the ingress gateway. Models update weekly; the classifier is 1.8 GB. The cluster is Intel Xeon with no GPUs, running Istio and Prometheus. Current Flask services have no batching, cold-start spikes of 3 seconds after deploys, and duplicate model downloads. Options include OpenVINO Model Server, Triton, and KServe. The team has OpenVINO expertise but limited Triton experience. Model formats: scikit-learn pickle, SavedModel, TorchScript. Compliance requires an audit trail of model versions and rollback within 5 minutes.
1. Recommend a serving stack and a conversion path for the three model formats. Justify the choice against the constraints.
Read the worked answer
Choose OpenVINO Model Server as the single serving layer, because it fits the team's existing expertise and supports OpenVINO IR, ONNX, and TensorFlow formats. Convert the TensorFlow classifier to OpenVINO IR with the Model Optimizer or openvino.convert_model, and convert the PyTorch reranker to ONNX or IR via torch.onnx.export and then OpenVINO. The scikit-learn demand model is not natively served by OVMS, so export it to ONNX with skl2onnx and serve that, or keep it behind a thin FastAPI sidecar if the pipeline uses unsupported operators. Standardize on a versioned model repository with immutable directories per model version. Deploy OVMS on Kubernetes behind Istio, with each model version as a separate deployment. This gives one runtime for two of three models and a documented exception for the third, which is better than adopting Triton solely for format coverage when the team cannot operate it confidently.
2. Design the rollout and autoscaling plan to meet the 120 ms p95 SLO and avoid three-second cold starts.
Read the worked answer
Separate the concerns. Use Istio weighted routing for canary releases: send 1%, then 10%, then 50% of traffic, checking p95 latency, error rate, and model output parity at each step. Keep the previous version warm for five-minute rollback. For autoscaling, use horizontal pod autoscaling on a custom latency or queue-depth metric rather than CPU alone, with a minimum of two warm replicas per model so a scale-from-zero does not create a cold start. Pre-pull model images and model artefacts into a node-local cache or persistent volume; 1.8 GB downloads are the real cause of the three-second spikes. Add readiness probes that load the model before the pod receives traffic. For the 200 rps peak, use OVMS dynamic batching with a small window, for example 2 to 5 ms, and test whether batching helps or hurts the p95. Set a pod disruption budget and roll out during low-traffic periods.
3. What would you do about the scikit-learn model and the compliance audit trail?
Read the worked answer
Do not force every model into the same runtime at the cost of reliability. Export the scikit-learn model to ONNX and validate numerical parity on a golden dataset; if the export fails or uses unsupported operators, serve it with a small, pinned FastAPI container that loads a versioned pickle from the same model repository. Either way, give it the same deployment, canary, and rollback controls as the other models. For audit, record model name, version, hash, conversion tool and version, training data snapshot, and the evaluation report in a registry, and emit the model version in every response header or log line. Rollback must be a routing change to a previously deployed immutable version, not a rebuild. Test rollback quarterly and measure it against the five-minute requirement; a documented runbook is not proof until it is exercised.