ML Solution Monitoring, Maintenance, and Security
Model and data drift
SageMaker Model Monitor captures endpoint data and compares it to a baseline: data quality, model quality (requires ground-truth labels merged later), bias drift, and feature attribution drift. Traps: expecting model quality monitoring to work without labels, or forgetting that monitoring schedules need a baseline produced by a baseline job. CloudWatch alarms on monitor violations drive automated retraining or rollback.
Operational monitoring
CloudWatch tracks endpoint latency, invocation errors, and instance metrics; Model Dashboard gives a fleet view. For retraining, Pipelines plus EventBridge can trigger on drift alarms, with Model Registry approval gating promotion.
Security and compliance
IAM execution roles scope what training jobs and endpoints can access; least privilege applies to S3 prefixes and Feature Store. Use VPC-only mode with private subnets, interface endpoints, and KMS encryption for data at rest and inter-container traffic. SageMaker Studio supports IAM Identity Center and per-domain isolation. Network isolation blocks outbound calls from training containers. Trap: granting broad S3 wildcards in execution roles, or assuming encryption is on by default for every artifact.
Auditing
CloudTrail records API calls, CloudWatch Logs capture container logs, and Model Registry lineage ties a deployed model to its data and parameters. Clarify and Model Monitor outputs feed governance reporting.
Worked example
A deployed credit model shows input feature distributions shifting. Rule of thumb: monitor data quality first, since label-free; schedule model quality with delayed labels. Configure a baseline job, a monitoring schedule, and a CloudWatch alarm that fires an EventBridge rule to retrain and require manual approval in the registry before redeployment.