Spot reclamation breaks the nightly batch three nights running

Refute or confirm with data before spending money on capacity - adding nodes makes the cost objection worse and will not stop reclamation.

Evidence to collect:

1. Timeline correlation. In Cloudera Manager, look at host charts for the failed nodes: a host whose CPU, memory and heartbeat series all stop at the same minute, with no preceding resource spike, looks like an infrastructure removal, not a workload problem. Skew produces one host pinned at 100% CPU for a long time and slow tasks, not series that flatline together.
2. Provider-side records. AWS Spot interruption notices and instance state-change events give you the exact time and reason per instance. Match them to the failure times. This is the decisive evidence.
3. YARN container diagnostics. Lost task attempts report the container as released on a lost node; the resource manager log records the node transitioning to lost. Count these per node group.
4. Distribution test. If failures cluster on a subset of hosts in the compute group at a small number of moments, and the On-Demand master nodes are unaffected, reclamation is the explanation. If failures track a specific table or key range regardless of host, it is skew.
5. Quantify: number of reclaimed instances per night, hours of work lost, and how many applications were retried successfully. That figure is what justifies the architecture change and sets the acceptance target for cost.

State the conclusion explicitly: this is an availability characteristic of the capacity model, not a data bug.

The principle: keep the interruptible capacity for work that is cheap to repeat, and put anything whose loss kills an application on stable capacity.

Changes:

1. Move application masters and drivers off Spot. Use a small On-Demand worker group and place Tez AMs and Spark drivers there, via YARN node labels or role configuration. Losing an executor costs one task attempt; losing the AM kills the application.
2. Shorten the unit of loss. Smaller task splits and more attempts per task mean a reclaimed node costs seconds of recomputation rather than a partition. Raise task attempt retries so YARN reschedules instead of failing.
3. Make the pipeline idempotent and restartable. Partition-level overwrite semantics and a workflow engine that retries failed steps without duplicating data. If a stage must be exactly-once, it belongs on On-Demand.
4. Handle the warning window. Spot reclamation gives a short termination notice; configure node decommissioning so running containers drain where the platform supports it, and tune the decommission timeout so it is shorter than the notice.
5. Right-size headroom. Keep enough On-Demand or diversified instance types that losing 10% of the compute group does not starve the queue.

What you accept: roughly a 5–15% throughput variance on bad nights, some retried work, and a longer tail. In exchange you keep most of the Spot discount. State that trade-off to finance in those terms. Buying 20% more Spot nodes without these changes just buys 20% more interruptions.

The monitoring failure here is more serious than the capacity failure: the outage was visible in the data for three nights and nobody was paged.

Alerting:

• Symptom-based, not host-based. Alert on workload outcomes: applications failing above a threshold, SLA completion at risk at a defined checkpoint (for example, 05:00 progress against expected partitions), and YARN pending containers sustained above a limit. Host-health emails detect hardware, not missed SLAs.
• Capacity-model signal. Track Spot interruption rate and lost-node count per night; a step change should page before jobs start failing.
• Routing and severity. Paging path for anything that threatens the batch SLA; the shared mailbox is a digest, not an alert channel. Name an on-call rotation and escalation.
• Dashboards per node group, so the On-Demand and Spot groups are compared directly.

Runbook:

1. Classify: are failures time-clustered across hosts (reclamation) or key-clustered (data)? Each branch has a different first action.
2. If reclamation: confirm remaining capacity, decide whether to re-run the affected applications immediately or defer to the morning, and notify the business owner with a revised ETA.
3. Recovery: identify failed applications, restart only the affected steps, verify output partitions before declaring success.
4. Record reclaimed instance count and lost work per incident, so the capacity decision is revisited quarterly with data.

The exam point: an alert that fires after the business notices is documentation, not monitoring.