Operations, Monitoring, and Governance of ML Jobs
What the domain tests
This domain covers running ML workloads reliably in production: scheduled jobs, job runs, logs, retries, alerts, quotas, cost, model and data monitoring, audit, lineage, and access governance. Expect questions on idempotency, failure handling, drift detection, and evidence for compliance.
Core rules
- Jobs should be idempotent. A retry must not double-insert rows or duplicate files; use merge or upsert, partition overwrite, or run identifiers.
- Monitor the job, not just the cluster. Track run status, duration, rows read and written, and error logs. Alert on failure and on silent anomalies such as zero rows.
- Separate training, scoring, and monitoring schedules. A nightly feature job, a weekly retrain, and daily drift checks have different SLAs.
- Monitor model performance and data drift. Label delay means you may need proxy metrics; compare incoming feature distributions with training baselines.
- Govern with Ranger, Atlas lineage, audit logs, and least-privilege service accounts. Every production model should trace to data, code, and a deployment version.
- Control cost with quotas, resource profiles, and retention policies for logs and artifacts.
Worked example
A nightly feature job fails at 02:00. The engineer checks the job run log, finds a schema change in an upstream Hive table, adds a schema validation step, configures one retry with backoff, and sets an alert to the on-call channel. They also add a row-count check that fails if the output is less than 80 percent of the prior day. Reasoning: the retry handles transient issues; validation catches breaking changes; the row-count check catches silent partial loads that still succeed.
Traps
- Retrying a non-idempotent append job and duplicating data.
- Alerting only on job failure, not on empty or stale outputs.
- Using an over-privileged service account for convenience.
- Monitoring accuracy without accounting for delayed labels.
- Keeping every experiment artifact forever and blowing the quota.
- Losing lineage between a deployed model and its training data.