The 92% Datastore at 08:15
You arrive at 08:15 to prepare a 24-seat instructor-led lab that starts at 09:00. Every seat runs a nested ESXi host on top of a shared three-node physical cluster, with one NFS datastore of 8 TB backing all 24 pods. Reverting a single pod to its clean snapshot takes six minutes, and a script can revert pods sequentially or in parallel batches. The address pool has 250 leases. At 08:15 you find the datastore at 92% used, three pods unreachable, and the previous evening's class log showing that 11 students deployed a second nested VM "to test something." Twenty-four students are enrolled, two walk-ins are expected, and one co-instructor who has never run this lab is available. The first exercise requires every pod to be reachable and clean.
1. Lay out your 08:15 to 09:00 triage plan in priority order, with the reasoning for the sequence and an estimate for the revert work.
Read the worked answer
1. Protect the shared resource first (08:15-08:28). At 92% of 8 TB there is roughly 640 GB free. A revert or a deployment on 24 pods can consume that in minutes and strand the whole room. Stop all pod activity, then find the top consumers: the log says 11 students created second nested VMs, and abandoned snapshot chains and orphaned VMDKs are the usual culprits. Delete only what you can positively identify as garbage; do not delete an unknown chain to buy space.
2. Restore reachability (08:28-08:35). Three pods down while 21 work suggests a specific node or a specific pod-level fault. Hand this to the co-instructor with a written decision rule: fix in eight minutes or the affected students get relocated to paired seats.
3. Verify pods in batches (08:35-08:55). Serial is impossible: 24 x 6 = 144 minutes. Batch six at a time for four batches - about 24 minutes plus overhead - but smoke-test one pod first to confirm the revert path still works after the cleanup. Watch the datastore graph during each batch, because reverts allocate before they free.
4. Seat the walk-ins (08:55). Decide their pod allocation after the batch results, not before.
2. What are your go/no-go criteria at 08:45, and what is the fallback if you fail them?
Read the worked answer
State the criteria before you start, so the decision is not made by fatigue or optimism.
Go requires all four: at least 20 of 24 pods revert-verified; datastore at or below 80%; both walk-ins assigned a working seat; and one demonstration pod that you have personally logged into.
Degraded go (18 or 19 verified): run the lab as a couplet exercise with 12 active pods, keep the pair's non-driver on the documentation task, and drop the optional fourth exercise. Announce it in the first minute with the reason and the changed end time.
No-go (fewer than 18, or the datastore cannot get under 85%): convert the first module to instructor-led demonstration with guided note-taking, postpone the hands-on to the afternoon after the platform team works the problem, and re-sequence the day so the no-lab content comes first. Notify the room before 09:00, not after they have logged in.
The governing principle: students forgive constraints that are stated early and paired with an alternative. They do not forgive discovering at 10:30 that the morning was unrecoverable.
3. What preventive changes would you propose so this does not recur, and which one would you push for first?
Read the worked answer
Push for capacity governance first, because every other control is weaker than simply not running out.
- Per-pod quotas and a reserved floor. Cap each pod so no single student can consume more than its share, and hold a floor of 20% free at all times. At 8 TB, 20% is 1.6 TB of protection that also absorbs the burst that reverts create.
- Automated end-of-day reversion. A scheduled revert of all pods after class removes the accumulated "test something" estate before it compounds. Run it nightly, not before class, so failures are discovered at 17:30, not 08:15.
- Snapshot and orphan alarms. Alert on any snapshot older than 12 hours and on datastore usage above 75%. The previous evening's misuse was visible and unmonitored.
- A written pre-class checklist with a T-45 go/no-go, including the datastore threshold, one successful test revert, and a named owner for each check. The co-instructor needs this document more than you do.
- Change freeze on the lab platform during teaching hours, with a named exception process.