I85PREP
⌘ K
CDESS study materials
CDESS study resources

CDESS: Data Engineering: CDE Jobs, Virtual Clusters, and Orchestration

CDESS · Work through your exam, one topic at a time.

All guide sections

Data Engineering: CDE Jobs, Virtual Clusters, and Orchestration

Data Engineering (CDE)

The Data Engineering domain tests whether you can define, schedule, and operate Spark workloads on Cloudera Data Engineering (CDE), the CDP service that runs jobs on Kubernetes virtual clusters rather than on static YARN queues. Expect questions on job types, resources, dependencies, run lifecycle, and how CDE sizing interacts with Spark configuration.

Jobs, resources, and repositories

  • A CDE job has a type: Spark, Airflow, Python, or Custom. Spark jobs reference a main application file plus arguments and Spark configuration; Airflow jobs reference a DAG; Python and Custom jobs run scripts or containers.
  • Files (JARs, Py files, configs) and Python environments are uploaded as CDE resources and mounted into the job pod. Referencing a path on the driver's local disk is a common failure.
  • Git repositories let job definitions come from version control; schedules are cron expressions interpreted in the cluster time zone.

Virtual clusters and sizing

A virtual cluster is backed by a Kubernetes compute cluster and has initial, minimum, and maximum compute settings plus autoscaling. Autoscaling adds pods only up to the configured maximum; it cannot exceed that ceiling. Driver and executor requests must fit within available capacity, so a job stuck waiting for executors usually means maxed-out resources or an unsatisfiable resource request.

Worked example

A nightly Spark job finishes at varying times, and a dependent aggregation job starts at 02:30 by cron. On a slow night the aggregation reads incomplete data. Correct fix: define the aggregation as a CDE job with a dependency on the upstream job, so it starts only after a successful run, and remove the second cron schedule. Reasoning: cron encodes a time assumption, while a job dependency encodes a data-ready condition, which is what correctness actually requires.

Traps

  • Assuming an Airflow job inherits the Spark configuration of the virtual cluster; each job carries its own configuration.
  • Treating autoscaling as unlimited capacity.
  • Expecting a failed parent run to still trigger a dependent job; the dependency waits for success.
  • Storing application files outside CDE resources and expecting them to be available in the pod.