A Contractor Firewall Change Takes Down ERP at Quarter-End
At 14:05 on the last working day of the quarter, the ERP system (Oracle DB, two application servers behind an F5 load balancer) becomes unreachable for 220 finance users. The service desk logs 90 calls in 10 minutes. The network team reports that a firewall rule was changed at 13:50 by a contractor to 'allow a quick test'; the rule blocked port 1521 between the app servers and the database. The change was made under an emergency ticket raised at 13:47, approved by the contractor's own team lead, and no backout plan was recorded. Service is restored at 14:55 after the rule is reverted. The ERP system is listed in the CMDB as a business-critical service with a 99.9% availability target. The quarter-end close is due at 18:00, and the finance director asks for a timeline.
1. Classify the records that must be created and explain how the incident, problem, and change records relate in this case.
Read the worked answer
Create one major incident for the ERP outage, not 90 separate incidents. The service desk should link the calls to the major incident so impact is visible without duplicating work. Record priority from impact and urgency: 220 finance users, a business-critical service and a quarter-end deadline make this P1. The incident record contains the timeline, the workaround (revert the rule), restoration time, and user confirmation.
The firewall modification is a change record. It should be an emergency change with the requester, implementer, approver, risk assessment, test evidence, implementation time and backout plan. Because the approval was not independent, the record should be flagged as a control failure, not simply closed as successful. The change record links to the incident it caused.
A problem record is also required. The incident restored service; the problem investigates why an untested, self-approved change could reach production and why detection took 15 minutes. The problem record holds the root cause, the contributing factors and the corrective actions. Do not close the problem when the incident closes.
The ERP service, its application servers, database, load balancer and firewall rule are configuration items in the CMDB. The change record should have referenced the ERP service relationship so the impact on port 1521 was visible before implementation. If the same failure recurs, a known-error record can link the symptom to the documented workaround. Keep the incident, change and problem records cross-referenced so the timeline can be reconstructed for the finance director and for the post-incident review.
2. Reconstruct the timeline and identify at least four control failures in the change process. Propose specific controls with owners.
Read the worked answer
Timeline. 13:47 emergency ticket raised; 13:50 firewall rule changed; 14:05 ERP unreachable; 14:05-14:15 service desk receives 90 calls; 14:55 rule reverted and service restored; 18:00 quarter-end close due. Detection took about 15 minutes and restoration 50 minutes from first report.
Control failures and controls.
- Self-approval. The contractor's own team lead approved the change. Control: emergency changes require an independent change manager or duty CAB approval, and the implementer cannot approve. Owner: change manager.
- No backout plan. The ticket had no rollback command or time limit. Control: the change template must require a tested backout plan, and implementation is blocked without it. Owner: network team lead.
- No impact analysis. The change did not reference the ERP service or the database port in the CMDB. Control: a mandatory CI impact report and peer review for firewall changes. Owner: service owner and network architect.
- No test or verification. The rule was changed directly in production with no staging test and no post-change check. Control: test in a lab or with a shadow rule, then run a synthetic login and monitor firewall denies after the change. Owner: network operations.
- Change timing and freeze. The change was made on quarter-end close, a known peak period. Control: a published change calendar and a freeze that requires director-level approval. Owner: change advisory board.
- Excessive contractor access. The contractor could change a production firewall rule without supervision. Control: least privilege, time-bound accounts, session recording and a named internal sponsor. Owner: security and procurement.
Each action needs a date and an owner; report status weekly until closed.
3. Define the post-incident review outputs and the metrics you would report to the finance director and IT management. Explain what each metric reveals.
Read the worked answer
Post-incident review outputs. A concise report with the timeline, business impact, root cause, contributing factors, what worked, what failed, and corrective actions with owners and due dates. Include the change record, the firewall rule diff, the service desk call log and the monitoring evidence. Add a communication summary showing when users and management were told what. Get the finance director to confirm the business impact and the close status.
Metrics for the finance director.
- Downtime and availability: 50 minutes of ERP unavailability. Against a 99.9% monthly target, a 30-day month allows about 43 minutes, so this single event already exceeds the monthly budget. Explain the calculation and the service credit or risk.
- Time to restore: 50 minutes from first report to service restoration. This shows how quickly the team recovered, but it does not show how quickly they detected the problem.
- Business impact: 220 users affected and the quarter-end close put at risk. State whether the close was completed and what manual work was needed.
Metrics for IT management.
- Time to detect: about 15 minutes, measured from the change to the first user call. This reveals that monitoring did not catch the failure.
- Change success rate and emergency change rate: the proportion of changes that caused an incident, and the proportion made outside the normal process. Both should fall.
- Percentage of emergency changes with an approved backout plan: directly measures the control that failed.
- Repeat incidents and problem closure rate: shows whether the corrective actions are real.
- Service desk first-contact resolution and call volume: 90 calls in 10 minutes shows a major-incident communication gap.
Report trends over three months, not a single point. Pair every metric with the action it justifies, and publish the action status until closure.