12,400 alerts a day and two outages found on social media
An online ticketing platform operates 4,100 configuration items. Its monitoring generates 12,400 alerts per day. About 91% are auto-closed within 90 seconds and roughly 3% are judged actionable by the on-call engineer. The on-call rotation is five engineers, two of whom resigned last quarter citing alert fatigue. Two real outages occurred last quarter. A payment callback queue failure lasted 30 minutes and a seat-map service failure lasted 95 minutes. Both were detected by customer posts on social media, not by the monitoring platform, although alerts for both had been raised and were buried in the stream. The director's response is to fund 200 additional synthetic checks. Separately, an engineer manually restarted a queue consumer at 02:00 without recording anything, and the identical failure recurred four days later.
1. Distinguish between monitoring and event management in this scenario. Where is the failure, and what evidence supports your diagnosis?
Read the worked answer
Monitoring is observation. It is the instrumentation, metrics, logs, traces and synthetic checks that produce data about the state of components and services. Event management is the sense-making layer: detecting events, filtering them, correlating them, judging significance, and deciding whether they should become an incident, a change trigger, or nothing at all.
The evidence points overwhelmingly at event management. The platform produces 12,400 alerts a day, of which about 372 are actionable. That is a signal-to-noise ratio of roughly one in thirty-three, which is not a coverage problem. Both missed outages had alerts raised. Detection worked; triage and routing did not. An alert that fires and is buried is functionally identical to no alert at all, except that it also consumes attention and erodes trust in the tooling.
The human evidence agrees. Two engineers resigned from a five-person rotation citing fatigue. The constraint in this system is not visibility, it is the finite attention of five people at night.
A useful diagnostic is to check whether every alert has a named owner and a documented action. Where an alert has no owner, it will be ignored by everyone. Also check whether alerts are tiered: a page to a human at 02:00, a ticket for the morning, and a dashboard for trend are three different destinations, and if all 12,400 go to the same place, the system has no notion of significance.
The unrecorded restart adds a second gap, but that belongs to incident management and problem management, not monitoring.
2. Evaluate the director's proposal to fund 200 additional synthetic checks.
Read the worked answer
The proposal treats a triage problem with a detection remedy, so it will make the symptom worse before it makes anything better. 200 extra checks added to a pipeline that already delivers 12,400 alerts a day with 3% actionability increases load on the exact resource that is failing: human attention in the on-call rotation.
Synthetic checks are not worthless. Journey-level checks that exercise the booking and payment flow from a user's perspective can detect failures that component metrics miss, and they can be written to produce one signal per user journey rather than one per endpoint. That is a legitimate use.
So the answer is a conditional yes with a much smaller scope: perhaps five to ten journey-level checks covering search, seat selection, payment callback and confirmation, each mapped to a service and each routed to a named responder with a documented action. Not 200 component checks.
The 200 checks should be deferred until the event pipeline is fixed. Sequence the work: first introduce alert tiers with distinct destinations, so pages go to a phone and everything else goes to a queue; second apply dependency-based suppression so one root component failure does not produce forty sibling alerts; third require every remaining alert to have an owner and a runbook, and retire anything that has not led to an action in 90 days; fourth measure the result. The measures are alerts per shift, pages per night, actionability rate, and the proportion of incidents detected by monitoring rather than by users. If actionability is 3% today, the target is not more coverage, it is a reversal of that ratio. Then revisit synthetic journey checks with the remaining budget.
3. A queue consumer was restarted manually at 02:00 and the failure recurred four days later. Which practices are engaged, and what would you do beyond telling the engineer to record it next time?
Read the worked answer
Four practices are engaged. Incident management requires a record of the failure and the action taken. Problem management requires that the recurrence be treated as evidence of an unresolved cause. Change enablement applies because an intervention in production was made, and if the restart was not pre-authorised it is an emergency change requiring retrospective authorisation. Knowledge management applies because the fix exists only in one person's head.
The risk of leaving it unrecorded is not administrative. The 02:00 restart and the four-day-later recurrence are the same event, and because neither was linked, the organisation lost the pair of data points that would have justified a problem investigation. A known error with a defined workaround could have made the second occurrence a two-minute fix rather than an open investigation. There is also a direct operational risk: a restart performed without record means nobody knows whether configuration was altered, so configuration records may now be wrong.
Telling the engineer to record next time is unlikely to work, and it may be unfair. Ask why the record was not made. Common causes are that the tooling is slow at 02:00, that recording the incident would trigger a morning interrogation, or that the action is not recognised as a change at all. Fix the cause: offer a one-line capture, or a chat-ops command that opens and closes an incident automatically.
Then act on the substance. Raise a problem record, link both occurrences, identify the failure signature, document a known error with the restart procedure, and add a health check and alert so the restart is triggered by the system rather than discovered by luck. If the restart becomes automated, it needs a change record and a log.