Month-End Spark Queue Starvation on a Mixed CDP Base Cluster
An 18-host CDP Private Cloud Base 7.1.9 cluster runs Hive, Impala, and Spark. Two masters run NameNode HA and the Hive Metastore; 16 workers each have 512 GB RAM, 64 vcores, and 12 x 4 TB HDD. HDFS is 64% full. YARN uses Capacity Scheduler with two queues: default 70% and etl 30%; preemption is disabled. During month-end, 30 Spark jobs are submitted. Each asks for 60 executors, 4 cores and 12 GB memory, plus 1 GB overhead. The default queue shows 18 applications pending; the etl queue runs. Cloudera Manager shows YARN memory utilization at 58%, vCore utilization at 96%, and many NodeManagers with 0 available vcores. Spark executors report long waits for containers, but there are no OOM kills. HDFS and Impala look normal. Management wants to add memory to the workers. You must decide whether that is justified and how to change the scheduler and YARN configuration without oversubscribing CPU.
1. What is the most likely bottleneck, and what evidence would you collect before approving more RAM? Explain why 58% memory utilization is misleading.
Read the worked answer
The immediate blocker is CPU/vcore allocation, not RAM. The queue is asking for 30 x 60 x 4 = 7,200 vcores at once, while the cluster has only 16 x 64 = 1,024 physical vcores; the default queue's 70% share is about 716 vcores. Memory demand is also large, but the observed 96% vcore utilization with 58% memory utilization indicates that containers are being packed until CPU is saturated while memory remains reserved but underused.
The 58% figure is misleading because YARN memory utilization measures allocated container memory, not physical CPU pressure or whether that memory is actively touched. A cluster can have plenty of allocated memory and still be unable to schedule work because no vcores remain. The per-vcore ratio also matters: 512 GB / 64 vcores is 8 GB per vcore, while these Spark executors request 13 GB / 4 cores, about 3.25 GB per vcore. The workload is CPU-heavy relative to the cluster's memory-to-CPU ratio, so adding RAM would make the imbalance worse.
Before buying memory, collect: yarn.scheduler.capacity.resource-calculator (DefaultResourceCalculator ignores vcores during allocation, while DominantResourceCalculator considers both), yarn.nodemanager.resource.cpu-vcores, yarn.nodemanager.resource.memory-mb, yarn.scheduler.maximum-allocation-vcores, spark.executor.cores, spark.task.cpus, RM scheduler metrics, yarn top, yarn queue -status, NodeManager container counts, and OS-level CPU steal/throttling. Also check whether physical cores are fewer than the configured vcores or whether cgroups are limiting CPU. Only after that can you decide whether the fix is more CPU, better vcore sizing, or a different executor shape.
2. Propose concrete YARN and Spark changes using the numbers in the setup. How do you protect the etl queue without preemption, and how would you roll this out safely?
Read the worked answer
First make CPU a real scheduling resource. Set yarn.nodemanager.resource.cpu-vcores=64 (not above physical cores unless you deliberately overcommit), yarn.nodemanager.resource.memory-mb to about 480 GB to leave OS headroom, and use DominantResourceCalculator. Set yarn.scheduler.maximum-allocation-vcores and maximum-allocation-mb high enough for one executor plus overhead, for example 16 vcores and 64 GB. Do not raise cpu-vcores to 128 just to make the queue shorter; that hides contention in the OS.
Because preemption is disabled, a queue with capacity 70 but maximum-capacity 100 can consume idle etl capacity and then block etl work when month-end arrives. Either cap the default queue's maximum-capacity at 70 so etl always retains its 30%, or enable preemption and accept that Spark containers may be killed. With preemption off, the safe choice is the cap. Add queue-level limits such as max-parallel-apps (for example 10 for default and 6 for etl), user-limit-factor=1, and a modest maximum-am-resource-percent so ApplicationMasters cannot crowd out executors.
Reshape Spark to the cluster ratio. Use spark.executor.cores=4, spark.executor.memory=28g, spark.executor.memoryOverhead=4g, and spark.dynamicAllocation.maxExecutors=12. That is 4 vcores and 32 GB per executor, about 8 GB per vcore, matching the worker ratio. One job uses at most 48 vcores, so roughly 10 to 14 jobs can run in the default queue instead of 30; the rest wait in waves rather than thrashing CPU. Roll out by changing one queue's Spark defaults, testing two representative jobs, and comparing queue wait time, container allocation, executor CPU utilization, and job SLA. Then expand. Keep the old executor settings for rollback.