Sixty Switches, One Four-Hour Window
You must push a firmware update to 60 branch switches. A 25-device pilot produced 3 failures: each dropped 802.1X and needed a physical console cable to recover, about 35 minutes apiece, one device at a time. In production, each upgrade takes 6 minutes and verification 4 minutes; a rollback takes 8 minutes plus verification. The approved window is Saturday 04:00–08:00. Fourteen of the 60 sites are 24×7 fulfilment centres with a 15-minute outage tolerance. Two engineers are on the call; each can run two upgrades concurrently, but console recovery is strictly serial. Leadership wants all 60 done this weekend.
1. Is "all 60 this weekend" achievable with two engineers? Show the arithmetic and name the binding constraint.
Read the worked answer
Window: 240 minutes.
Nominal throughput is not the problem. Four parallel streams (2 engineers × 2 concurrent) at 10 minutes per device yields 4 × 24 = 96 device-slots. Sixty devices consume 60 × 10 / 4 = 150 minutes, leaving 90 minutes of slack.
Recovery is the problem. Pilot failure rate is 3/25 = 12%, so expect 60 × 0.12 ≈ 7 devices needing console recovery. At 35 minutes each and strictly serial: 7 × 35 = 245 minutes - more than the entire window, before counting the 53 successful upgrades or any rollback. Even at an optimistic 5% failure rate, 3 recoveries consume 105 minutes, which is 44% of the window serialised through a single resource.
Binding constraint: the single-threaded out-of-band recovery path, not upgrade capacity. Secondary constraints: the 14 24×7 sites, whose 15-minute tolerance is shorter than the 35-minute recovery time - one failure there is a breach before you can even reach the console.
Verdict: not feasible as stated. The arithmetic only works if the failure mode is engineered out first.
2. Design the rollout you would actually approve and state your abort criteria.
Read the worked answer
1. Fix the failure before scaling. A 12% pilot failure rate is a stop signal, not a variance. Root-cause the 802.1X drop (config drift, certificate, RADIUS timeout) and re-pilot ~10 devices to under 2% before touching 60.
2. Remove the serialisation. Terminate console access to a console server / OOB path so recovery does not require a person and a cable at the site.
3. Sequence. Exclude the 14 24×7 sites entirely - their 15-minute tolerance is below the recovery time, so the expected value of including them is negative. Do 24–30 non-critical sites in this window.
4. Reserve a rollback buffer. Stop starting new upgrades at T+180, keeping 60 minutes for rollbacks and verification.
Abort criteria, agreed in writing before the window: more than two consecutive failures; any impact to a 24×7 site; any device unreachable 15 minutes after upgrade start; RADIUS authentication latency outside baseline; engineer count dropping below two. On abort, restore the last known-good config set and document the stopping point.
3. Leadership offers a second pair of engineers for the final two hours. Does that change your answer?
Read the worked answer
Partially, and not in the direction leadership expects.
- What it fixes: a second recovery-capable engineer halves the serial recovery burden, from 245 minutes to about 123 - which does make 60 devices arithmetically reachable at the observed 12% failure rate.
- What it does not fix: the failure rate itself. You would be running the 24×7 sites against a 15-minute tolerance with a 35-minute known recovery, betting on a coin-flip you have already seen land wrong three times in twenty-five. Recovery may also slow under two simultaneous recoveries competing for the same skills and the same runbook.
- What it does not fix either: a 12% failure rate usually indicates one shared root cause. More hands does not reduce the probability of a fourth, fifth and sixth failure.
Decision: accept the engineers, still do not do all 60. Use the added capacity to enlarge the non-critical wave to about 40 and to pre-stage OOB on the remainder, keeping the 14 24×7 sites out until the failure rate is under 2% in a re-pilot.