Deployment and Orchestration of ML Workflows
Inference options
Know the four serving patterns: real-time endpoints (persistent, low latency), serverless inference (scale to zero, cold starts, payload and timeout limits), batch transform (large datasets, no endpoint), and asynchronous inference (large payloads, long runtimes, near-real-time with S3 input/output). Multi-model endpoints host many models behind one endpoint and load them on demand; multi-container endpoints run several containers on one endpoint. Traps: choosing real-time for sporadic traffic where serverless is cheaper, or batch transform when the caller needs an immediate response.
Deployment strategies
SageMaker supports all-at-once, canary, and linear traffic shifting, plus automatic rollback triggered by CloudWatch alarms. Shadow or A/B testing compares variants. For edge devices, SageMaker Edge Manager and Neo compile models for target hardware. Trap: expecting automatic rollback without an alarm attached to the endpoint.
Orchestration
SageMaker Pipelines define steps (processing, training, tuning, evaluation, conditional, register) as a DAG; Step Functions coordinates broader multi-service workflows; EventBridge schedules and triggers runs. Pipeline parameters and caching reduce cost and rerun time. Trap: confusing Pipelines with Step Functions when the workflow spans non-SageMaker services.
Worked example
A nightly scoring job processes 40 GB with 30-minute tolerance. Rule of thumb: no persistent endpoint needed. Use batch transform on a schedule, or asynchronous inference if per-record latency matters. Wire it with EventBridge to a Pipeline whose evaluation step gates registration.
Cost and scaling
Right-size instance types, enable autoscaling on invocations per instance, and use managed warm pools or provisioned concurrency only where latency requires it.