Enterprise AI on Cloudera Machine Learning
The Enterprise AI domain tests how Cloudera Machine Learning (CML) supports the model lifecycle on CDP: experimentation, reproducible environments, deployment, and governance. Questions often ask where a step happens-project, session, job, model, or deployment-and how AI artifacts inherit lakehouse security.
Core concepts
- CML is a Kubernetes-based data science service in CDP. It provides projects (Git-backed), sessions (interactive compute engines), jobs (scheduled or triggered runs), experiments (tracking parameters and metrics), models (registered artifacts), and deployments (REST endpoints).
- Engines are containerized runtime templates (Python, R, Scala, Spark) with resource profiles for CPU/GPU and memory. Sessions are ephemeral; jobs are for repeatable pipelines; deployments run model replicas behind an endpoint.
- AMPs (Applied ML Prototypes) are end-to-end use-case templates with code, data, and instructions. They accelerate starts but are not automatically production-hardened.
- Governance: CML integrates with SDX. Ranger controls data access; Atlas captures lineage from data to model; the model registry versions artifacts; deployments run under a defined identity and can be monitored for drift in supported versions.
- Reproducibility: projects track code via Git; engines pin dependencies; experiments log parameters and metrics; model builds capture artifacts and environment.
Worked example
A data scientist must train on a Hive table containing PII and deploy a REST endpoint without exposing raw PII. Use a CML project; create a Spark session; read the Hive table under Ranger-authorized access; mask or tokenize PII in the feature pipeline; log the experiment; register the model; deploy it as a model endpoint. Atlas records lineage from Hive table to model; Ranger enforces column masking for other users. The endpoint does not bypass governance because it runs under a defined identity.
Traps
- Thinking CML sessions persist indefinitely; they are ephemeral and resource-bounded.
- Confusing model registration with model deployment; registering does not automatically create a serving endpoint.
- Assuming AMPs are production-ready; they need review, data substitution, and testing.
- Ignoring that Git-backed projects are the source of truth; manual edits outside Git break reproducibility.
- Believing CML bypasses Ranger because it is a data science tool; all data access is governed.