Cloudera Machine Learning Engineer common mistakes
Cloudera Machine Learning Engineer · Understand the traps and how to avoid them.
Treating a Cloudera AI (CML) project as a plain Git checkout and assuming that anything written outside the project's mounted files are persists after the session stops, then losing trained model artifacts, notebooks, and intermediate data when the session is restarted.
Candidates come from local Jupyter or cloud notebook habits where the container filesystem feels durable. In CML, only the project directory (and explicitly mounted data paths) is durable; the rest of the container, including most of the home directory and /tmp, is ephemeral and rebuilt on session restart or engine change.
What to do instead
Write all outputs that must survive into the project directory (or an external durable store such as HDFS, S3, or a database), commit code and small artifacts to Git, and register large artifacts in the project or a model registry. Before finishing any lab, restart the session once and confirm the artifacts are still reachable. Know which paths are project-relative versus absolute.
Choosing an engine/kernel and resource profile arbitrarily, then blaming the platform when a job fails with out-of-memory or missing-library errors, or when code that ran in an interactive session fails in a job run.
Candidates do not map the workload to the engine: a GPU profile is needed for deep learning, a larger driver/executor for Spark, and a specific Python version for pinned dependencies. They also forget that an interactive session and a scheduled job may use different engines, so installed packages and environment variables are not automatically identical.
What to do instead
Decide the engine from the workload (Python version, GPU vs CPU, Spark vs plain Python), size the resource profile from observed memory/CPU use, and pin dependencies in a requirements file or project environment so sessions and jobs resolve the same environment. Reproduce the job's engine and profile in an interactive session when debugging.
Using local filesystem paths or pandas-style reads against data that actually lives in HDFS, S3, or a relational database, or hard-coding credentials and connection strings in notebooks.
Candidates default to pd.read_csv('/data/file.csv') from tutorials. In a governed cluster, data is reached through HDFS, object storage, Hive/Impala tables, or JDBC, and access requires the right credentials and permissions. Hard-coded secrets also break when the project is shared or promoted.
What to do instead
Read through the supported connectors and URI schemes (hdfs://, s3a://, JDBC with a driver, or a SQL query against a Hive/Impala table), and keep credentials in project environment variables, the platform's secret handling, or a credential provider rather than in code. Verify the identity the session runs as and that it has read permission on the path or table.
Skipping data validation and leakage checks: training on raw data with nulls, duplicates, or target leakage, and evaluating on a random split of time-ordered data.
Candidates focus on model accuracy and treat preparation as a formality. Leakage from a random split of temporal data, or from features computed using the full dataset before splitting, produces impressive offline scores that collapse in production.
What to do instead
Build an explicit preparation stage: profile the data, handle nulls and duplicates deliberately, and split before fitting any transformation (fit scalers/encoders on train only, or use a pipeline). For time series, split chronologically and validate on a later window. Document the checks so they run again on new data.
Comparing experiments without controlling variables: changing data, features, hyperparameters, and code all at once, and recording results only in notebook output that is later overwritten.
Candidates iterate quickly and rely on memory or the last cell output. Without tracked runs, they cannot attribute a metric change to a specific change, and results are not reproducible when the session is restarted.
What to do instead
Track each run with its parameters, metrics, code version (Git commit), and data version, using the platform's experiment tracking or an equivalent. Change one factor per run, keep a baseline, and log metrics to a durable store rather than only printing them. Be able to reproduce the best run from its recorded inputs.
Reporting a single aggregate metric on one split and declaring the model ready, without a holdout, cross-validation, or inspection of per-class or per-segment performance.
Candidates optimize the headline number and miss that a high average can hide poor performance on a minority class or an important segment, and that tuning on the test set inflates the estimate.
What to do instead
Use a train/validation/test discipline or cross-validation, tune only on validation data, and report the final number once on the untouched test set. Add per-class and per-segment breakdowns and a confusion matrix where relevant, and state the metric that matches the business cost of errors.
Deploying a model by pointing a serving endpoint at a notebook that loads data, trains, and then predicts, or by assuming the endpoint will re-run training code.
Candidates conflate the experimentation notebook with the deployment artifact. A serving endpoint must load a persisted model and return predictions for a request; it must not depend on training-time data or long-running setup that will time out or fail at request time.
What to do instead
Separate training from inference: persist the fitted model (and its preprocessing pipeline) as an artifact, and write a minimal predict function that loads that artifact and transforms a single request consistently with training. Test the endpoint with sample requests, including edge cases, before declaring it live.
Ignoring serving constraints such as cold start, request timeout, concurrency, input schema, and resource sizing, so the endpoint works in a manual test but fails or slows under real traffic.
Candidates test with one request from a notebook and never consider that the serving pod may need to load a large model on first call, that requests arrive concurrently, or that the input will not match the training schema.
What to do instead
Validate the input schema and return clear errors for malformed requests, keep model loading efficient, size the endpoint's resources for expected concurrency, and test with multiple simultaneous requests and realistic payloads. Document the request/response contract and monitor latency and error rate after deployment.
Scheduling a job without handling failure and idempotency: no retries, no alerting, and a job that appends duplicate rows or overwrites good output when re-run.
Candidates treat a scheduled job as a one-off script. In operations, jobs fail on transient errors, run late, or are re-run manually, and a non-idempotent write corrupts downstream data while nobody notices because no alert fired.
What to do instead
Design jobs to be idempotent (partition overwrite, merge/upsert, or a run marker), configure retries and timeouts, and route failures to alerts or a monitored channel. Record run status and outputs so a failed run is visible, and test a deliberate re-run to confirm no duplication.
Treating governance as someone else's job: copying sensitive data into a personal project or local file, granting broad access to make a pipeline work, and leaving no record of who trained which model on what data.
Candidates prioritize getting the pipeline green and bypass restrictions by widening permissions or duplicating data. This creates compliance exposure and makes audits impossible, and it often breaks later when the broad grant is revoked.
What to do instead
Access data in place under the identity that already has least-privilege access, request scoped permissions rather than broad grants, and avoid copying sensitive data to unmanaged locations. Keep lineage from data source to model version, document purpose and owner, and confirm that monitoring and retention practices match the governance policy.