A 24x7 Monitoring Service with Six Operators and a Regulator Deadline
Qingshui Water serves 1.2 million customers through 380 remote pumping and pressure stations linked over commercial 4G. Monitoring today is 8x5: one engineer watches a dashboard and carries a pager at night. The regulator now requires detection of pressure loss within 15 minutes and a written record of response; two near-misses last year went unnoticed for over three hours. A 24x7 monitoring service must go live in five months with a staffing budget for six operators. The vendor supplies dashboards and alarms but no SLA. Station modems share a single credential with the corporate network, which is flat. Nobody has defined what 'the service is up' means, what happens when a station's 4G link drops, or who declares a major disruption. Operations wants to launch first and formalise later; the regulator disagrees.
1. List the elements that must be agreed in the service design package before go-live, and for each explain what fails if it is skipped.
Read the worked answer
The Service Design Package must be signed before the pilot, not after.
| Design element | Must state | What fails if skipped |
|---|---|---|
| Service definition and outcome | 'Detection of abnormal pressure at any of 380 stations within 15 minutes, 24x7, with a retained response record' | The regulator's requirement is never translated into a testable target; 'the service is up' stays undefined |
| SLA with the business, OLA with the 4G provider and field maintenance | 15-minute detection is only achievable if link-fault response is under 30 minutes; today field response is 4 hours | You commit to something no supplier has committed to |
| Availability definition | Per-station versus aggregate, planned versus unplanned, whether a powered-down station counts | Every outage becomes an argument and the percentage is unmeasurable |
| Capacity | Alarm rate per shift and behaviour in a storm (hundreds of alarms in minutes) | Operator is flooded; genuine alarms lost - exactly the near-miss failure mode already seen |
| Continuity | Monitoring-centre power and comms loss, fallback pager tree, manual watch | The watch stops precisely when the most stations are failing |
| Security | OT/IT segregation, per-station credentials, no shared corporate login | One compromised modem exposes the corporate network, and the audit trail cannot evidence who acted |
| Reporting and retention | What the regulator receives, in what format, kept how long | Evidence exists but cannot be produced on demand |
Moving from 8x5 to 24x7 is a new service, not an extended rota. Signing one package containing these seven items is what makes the launch defensible.
2. Is a budget of six operators sufficient for a 24x7 single-seat monitoring service? Show the arithmetic and state what you would do with the answer.
Read the worked answer
Arithmetic. A continuously staffed seat needs 168 hours a week. At 40 hours per FTE that is 4.2 FTE before any absence. Add realistic absence - 20 days leave, 5 days sickness, 5 days training, about 12% - and the requirement becomes 4.2 / 0.88 ≈ 4.8 FTE. Six operators give 240 contracted hours, roughly 211 after absence: a ratio of 1.26 against the 168 needed.
Verdict: six is the floor, not a comfortable position. Three specific gaps:
- No second seat during an alarm storm - precisely when detection matters most.
- No shift lead separate from the seat, so quality review and regulator reporting happen in overtime and eventually stop happening.
- Any absence spike above 12% forces the on-call engineer to cover the seat, reintroducing the current failure mode under a new name.
What to do: present the rota model to the sponsor with three costed options - 7.5 FTE (six operators plus a shift lead/relief), six FTE plus alarm-triage automation to cut volume, or six FTE with contracted overflow cover from the field maintenance provider. Compare each against regulatory penalty exposure and the cost of one missed pressure-loss event. A 0.5 FTE shortfall is cheap next to an enforcement notice. Do not absorb the gap as overtime: it hides the shortfall until attrition exposes it, at the worst possible moment.
3. Operations wants to launch and formalise later. Define the minimum gate you would not cross, and how you would phase the launch to satisfy both sides.
Read the worked answer
'Launch first, formalise later' is defensible for documentation and indefensible for the alarm path. Draw the line at what the 15-minute requirement actually depends on.
Irreducible pre-launch gate (on a 40-station pilot):
- End-to-end alarm path tested from a real station, including the case where that station's 4G link is down and the alarm must arrive from the polling layer instead.
- Written definition of the 15-minute clock - detection only, or detection plus operator acknowledgement - agreed with the regulator in advance. This single ambiguity can invalidate the entire compliance claim.
- Named authority to declare a major disruption, plus an escalation tree exercised once, at night.
- Immutable record of alarm, acknowledgement and action times, exportable without vendor help.
- Credential separation between station modems and the corporate network. This is not a documentation item; it is the difference between a monitoring outage and a network compromise.
Deferrable: full service catalogue entry, dashboard polish, trend reporting, a formal CSI register.
Phasing to sell it: pilot 40 stations with the gate artefacts complete, regulator walkthrough at day 60, then scale to 380 while documentation matures in parallel. Operations gets an early launch; the regulator gets inspectable evidence for the stations in scope. Launching all 380 on an untested alarm path risks a false assurance claim - a worse regulatory outcome than a two-month delay.