The Inherited Monitoring Dashboard
You have just joined a 12-person development team responsible for a payment service that processes about 200,000 transactions per day. The team inherited a monitoring dashboard that currently fires roughly 40 alerts per day. When you ask, the on-call engineer tells you that most of these alerts are ignored, and that the team has not had a real outage detected by the dashboard in over a year. You also learn that the dashboard was built around a set of thresholds chosen at launch, and that the service has since grown by roughly a factor of five. The team lead asks you to improve the situation, but says there is no budget for new observability tooling this quarter, and that the whole effort must fit within one engineer's spare time. You have three weeks before the next on-call rotation.
1. How would you diagnose whether the alert noise is a threshold problem, a design problem, or a coverage problem, and what would you look at first?
Read the worked answer
The first step is to separate the three possible causes with evidence rather than intuition. I would pull the last 30 days of alert history and classify each firing into three buckets: (a) alerts that led to a real, user-visible incident; (b) alerts that fired but were dismissed as expected behaviour; (c) alerts that fired and were never acknowledged. Bucket (c) is a strong signal of design problems such as missing runbooks, alerts on causes rather than symptoms, or alerts that do not tell the responder what to do. Bucket (b) is a strong signal of threshold problems, because the threshold no longer reflects normal behaviour after the fivefold growth. Bucket (a), and especially any incident not detected by the dashboard, is the coverage problem.
I would look at the dismissed alerts first because they are the cheapest to fix and the most likely to be threshold-related given the growth. For each dismissed alert I would compare the current baseline to the configured threshold and check whether the threshold was set in absolute units that no longer match traffic. I would then look at the unacknowledged alerts to see whether they are duplicates, whether they lack an owner, or whether they fire on a metric that nobody can act on. Only after that would I examine incidents missed by the dashboard, because coverage gaps are usually the most expensive to close and should be prioritised against the actual risk they represent. The three weeks are best spent on the dismissed and unacknowledged buckets, with a short written list of coverage gaps handed to the team lead for later.
2. Given the constraints, what concrete changes would you make in three weeks, and how would you decide what not to do?
Read the worked answer
I would treat the work as a small, reversible experiment rather than a monitoring rebuild. In week one I would fix or silence the dismissed alerts: for each one, either raise or convert the threshold to a rate or ratio that scales with traffic, or delete it if it has never led to action. I would also add a one-line owner and a first action to every remaining alert's description, because an alert without an owner is just noise. In week two I would consolidate duplicate alerts into a single symptom-level alert per user journey, and I would add a small number of genuinely missing symptom alerts for the payment path, such as a drop in successful transaction rate or a rise in latency at the payment gateway. In week three I would run a short review with the on-call engineer and measure the change.
The decision about what not to do follows from the constraint. New tooling is out of scope. Rewriting the whole alert catalogue is out of scope. Anything that requires a change to the service's instrumentation is out of scope unless it is trivial, because that would need development time the team does not have. I would explicitly not chase coverage for every possible failure mode; instead I would rank gaps by the product of likelihood and user impact and only close the top few. I would also not promise a permanent fix. The deliverable is a measurably quieter dashboard, a documented list of remaining gaps, and a short note to the team lead explaining which gaps need budget or development time in the next quarter. That keeps the effort honest and prevents the team from mistaking a quieter dashboard for a safer service.