Two senior engineers leave mid-upgrade at a utility
A regional energy utility runs six Multi-Domain Security Management domains covering 55 gateways, split between corporate IT and four operational regions. Two senior engineers who designed the environment resigned within a month of each other, mid-way through a management upgrade that has now stalled with two domains upgraded and four not. The remaining team is three administrators with around eighteen months' experience each, none certified beyond the entry level. Mean time to resolve priority-one incidents has risen from under four hours to just over two days, and the operations director has two concerns: the stalled upgrade and the fact that the board's availability commitment is now being missed. Support is currently on the vendor's base tier, and the training budget was cut in the last cycle. The next regulatory audit requires evidence of trained, competent administrators and documented change control.
1. Recommend how to use Check Point support and services to recover, and justify the support tier change with a cost argument.
Read the worked answer
Recover in this order: stabilise, then finish the upgrade, then prevent recurrence.
Stabilise: open a priority case describing the mixed-version environment explicitly, because a half-upgraded management estate is a known-risk state and the vendor's guidance on domain upgrade order matters more than speed. Ask for a named technical contact rather than a queue, since two days to resolve priority-one is a symptom of context being re-explained on every call.
Finish the upgrade: buy a fixed-scope professional services engagement for the remaining four domains, with the goal stated as knowledge transfer, not merely completion. Have the three administrators shadow the work and own the runbook, otherwise you buy the same problem again at the next upgrade.
Tier argument: compare the current tier against the cost of the availability commitment being missed. Two days of priority-one resolution against a four-hour target, repeated across a quarter, is measurable in contractual penalties and in audit findings; the higher support tier and a design review are almost certainly cheaper. Frame the ask as buying a named contact, proactive health checks on the management estate, and faster response targets, and require the vendor to state those commitments in writing so the board can hold someone to them.
Do not buy more licences as a substitute for capability.
2. Build a training and certification plan for the three remaining administrators, sequenced over twelve months and defensible to the auditor.
Read the worked answer
Sequence by risk ownership, not by enthusiasm for certificates.
Months 1-3: all three complete foundation-level training (CCSA) and, in parallel, each takes ownership of one domain as a documented runbook. The runbook is what the auditor actually wants to see; the certificate alone proves attendance.
Months 4-8: two administrators progress to expert-level training (CCSE), because they are the ones who will touch policy and upgrades. The third takes the operational track focused on monitoring, logging and incident handling, which matches the mean-time-to-resolve problem better than another architecture course. Add a short, vendor-agnostic change-control module delivered internally, since the stalled upgrade is as much a process failure as a skills failure.
Months 9-12: one administrator starts the expert or managed-security track (CCME level) as the designated deputy for the environment, with a written succession note. This is the single most important line item: with two departures in a year, the environment currently has no successor.
Justification to the auditor: a training matrix mapping each administrator to the domains they operate, dates completed, and the runbooks they authored; plus evidence that change records exist for the upgrade and for the abandonment of the window. Restore the budget by showing the audit finding risk, not by arguing that training is good practice.
3. The upgrade is stalled and the team is afraid to continue. Describe how you would decide whether to finish or roll back, and what process changes follow either way.
Read the worked answer
Decide on evidence, not on confidence. First establish the actual state: which domains are on which version, whether the upgraded domains are stable under load, whether the management server can still push policy to the unupgraded domains, and whether rollback is even possible without losing policy changes made since the first domain was upgraded. That last question often settles the decision, because a rollback that costs a week of policy work is worse than a controlled completion.
If the upgraded domains have been stable for a defined observation period and policy distribution still works, finish the upgrade in a scheduled sequence, one domain per window, with a tested rollback point before each. If either condition fails, stop, freeze changes, and roll forward to a supported state rather than backwards; a partially upgraded estate held indefinitely is the worst of the three options.
Process changes either way: a named change owner for the management estate; a pre-upgrade checklist including backup verification and a written rollback trigger; peer review before any window; and a post-incident review that records why the work stopped, because the current situation has no documented reason and that is why the team is afraid.
Report the decision and its rationale to the operations director in writing. The audit will ask who decided and when.