Faster alerts, slower recovery at Northbridge Bank
Northbridge Bank runs card authorisation across a 46-node estate. At 08:12 on Monday the monitoring platform raised 341 alerts in six minutes across 22 configuration items. Nine service desk analysts were on shift, two of them in their first month. First line matched and closed 188 alerts as duplicates, declared a P2 at 08:31, and escalated to the application team at 08:44; the application team joined the call at 09:02. Card declines peaked at 12% of authorisation traffic between 08:14 and 09:26. The cause was a weekend batch job that had not released database connection pool connections; the pool was cleared at 09:26. A March review recommended faster detection, so the monitoring team added 60 threshold alerts. Mean time to acknowledge has fallen from 4 minutes to 90 seconds, while mean time to restore has risen from 41 minutes to 68 minutes over the same period. The CTO has asked why.
1. Explain how adding 60 threshold alerts could improve acknowledgement time while degrading restoration time. Which practices are involved, and where is the actual failure point?
Read the worked answer
The two metrics measure different parts of the value stream. Acknowledge time stops at the moment a ticket exists; restore time starts at the same point and ends when service is usable. Adding threshold alerts shortens the first because each alert is itself a ticket trigger. It lengthens the second because the number of human judgements required before diagnosis rises.
The important observation is arithmetic. 341 alerts across 22 configuration items in six minutes is almost certainly one causal event with a fan-out through dependent components, not 341 independent failures. The monitoring and event management practice exists precisely to turn raw events into a small number of meaningful management signals, using filtering, correlation, deduplication and significance rules. Here that work was delegated to nine analysts, two of them newly trained, under peak pressure. 188 alerts were closed as duplicates, which is a manual, error-prone version of what correlation should have done automatically.
The failure point is not detection. Detection worked: the platform saw the pool problem within two minutes. The failure is in the transition from alert to declared incident. A P2 nineteen minutes after the first alert, at a point when 12% of card authorisations were failing, suggests prioritisation was driven by ticket category rather than by business impact. A revenue-affecting, customer-visible failure across multiple teams meets typical major incident criteria and should have triggered a different response: one parent incident, a single technical bridge, an appointed major incident manager, and stakeholder communication.
The lesson is that alert volume is not a proxy for monitoring quality. The team optimised the measure the CTO could see rather than the outcome the customer experiences.
2. It is 08:14. You are the duty service desk manager. Identify the specific decisions and actions that should occur before 08:30, and state who owns each.
Read the worked answer
At 08:14 the first decision is significance, not ownership. Someone must ask what these alerts have in common and whether they represent one failure. With two alerts on the same database pool and a growing count of dependent CI alerts, the correlation answer is available within a minute or two from the dependency data already held in the configuration management system.
The second decision is impact. Authorisation success rate is the only number that matters, and it should be visible on a service health view rather than reconstructed from tickets. If the decline rate is above roughly 5%, this is a major incident regardless of how many components are involved.
The third decision is declaration and role assignment. By 08:20 the duty manager should declare a major incident, name a major incident manager who is not also triaging tickets, and open one technical bridge with named leads from application, database and monitoring. The major incident manager owns coordination and communication; the technical lead owns diagnosis. Never the same person.
The fourth decision is containment of work. Individual analysts stop matching and closing duplicates; all alerts are linked as children to a single parent incident so the timeline stays intact.
The fifth is communication. Customer service and the payments business need a holding message with known impact and next update time, even if the cause is unknown.
The sixth is authority. Somebody must be empowered to approve an emergency change, such as releasing pool connections, without waiting for a normal change cycle. If the duty manager cannot authorise that, the escalation path should be a phone call, not a ticket.
3. Recommend three changes to the monitoring configuration and explain how you would measure whether they worked.
Read the worked answer
First, replace per-component thresholds with service-level signals. Instrument the authorisation success rate and the 95th percentile response time, and alert on those. Component alerts should enrich the service-level signal rather than summon people independently. Measure this by counting alerts raised per incident: today the ratio is roughly 341 to one; a target of five or fewer is realistic.
Second, implement dependency-based suppression. If the configuration management system knows that 22 CIs sit behind one database pool, an alert on the pool should suppress or aggregate the dependent alerts. Measure suppression effectiveness by the proportion of alerts that reach a human as part of a correlated group, and by the number of distinct pages per on-call shift. A useful companion measure is the proportion of closed alerts that were closed as duplicates, which today is 55%.
Third, automate the known response. Pool exhaustion has a known remediation. A runbook-triggered automation can release connections, log the action, and raise a low-priority incident record for review. Measure automation coverage and automation success rate, and keep a count of interventions that still require a human at 02:00.
Across all three, track detection source. The share of incidents detected by monitoring versus reported by users or found by chance is the honest test of whether monitoring is effective. Alongside it, track mean time to restore and repeat incidents within 30 days. If alert counts fall but restoration time does not, the work has only moved the noise. Also retire alerts deliberately: every alert that fires should have a documented action and a named owner, and any alert that has never led to an action in 90 days should be removed or downgraded to a dashboard.