I85PREP
⌘ K
JCPA study materials
JCPA study resources

Event-Driven Remediation Loops on a Flapping Link

JCPA · Put your knowledge to work in context.

All case studies

Event-Driven Remediation Loops on a Flapping Link

A team deploys an event-driven automation app: when a syslog/telemetry event indicates an interface went down, a Python handler runs a playbook that disables the interface and opens a ticket. During a maintenance window, a link flaps 40 times in 10 minutes (expected, due to fiber work). The handler fires 40 times, opens 40 tickets, and disables the interface - which the maintenance team then has to re-enable manually. The app has no deduplication, no rate limiting, and no notion of maintenance mode. The team wants to add 'if event seen in last 5 minutes, ignore' and call it done.

1. Explain why a simple 5-minute dedup window is insufficient and what the handler should check first.

Read the worked answer

A 5-minute dedup window reduces ticket spam but does not address the wrong action: disabling an interface during planned fiber work is harmful regardless of how many times it happens. The handler should first check context:

  1. Maintenance mode - is the device/interface in a declared maintenance window? If yes, suppress remediation (still log).
  2. Event severity and type - a link-down during maintenance is expected; a link-down outside maintenance may warrant action.
  3. Current state - is the interface already down/admin-down? Re-disabling is a no-op that hides the real state.
  4. Blast radius - is this a single link or many? Many simultaneous downs suggests an upstream event, not 40 independent faults.

Dedup is a rate control; maintenance mode is a correctness control. You need both, and correctness comes first.

2. Design the remediation logic so it is safe under flapping and still useful for real faults.

Read the worked answer

Proposed logic:

  1. Suppress if the interface/device is in a maintenance window (check a source of truth, e.g., a maintenance calendar or a maintenance: true tag). Log the event, do not act.
  2. Deduplicate by (device, interface, event-type) key with a short TTL - but treat this as throttling, not as the decision.
  3. Debounce: require the interface to be down for N consecutive evaluations or M seconds before acting, so a single flap does not trigger.
  4. Escalate, don't disable, on repeated flaps: if the same interface flaps more than K times in a window, open one ticket and page, rather than disabling. Disabling is a last resort for a confirmed persistent fault.
  5. Idempotent action: check current state before changing it; record the action taken.
  6. Kill switch: a global flag to disable all remediation during broad maintenance.

This makes the app safe during planned work and still responsive to genuine, persistent faults.

3. How would you test this logic before trusting it in production?

Read the worked answer

Test at three levels:

  1. Unit tests for the decision function: feed synthetic event sequences (single flap, 40 flaps in maintenance, 40 flaps outside maintenance, persistent down) and assert the action (suppress / debounce / ticket / disable). This is where the maintenance-mode and debounce logic is proven.
  2. Replay real historical events (including the maintenance-window incident) through the handler in dry-run mode and compare actions to what a human would do.
  3. Canary on a small set of non-critical devices with remediation in dry-run first, then enabled, with a kill switch and monitoring on action count.

Only after the replay shows zero harmful actions in the maintenance scenario should remediation be enabled broadly. The judgement: the incident is a test case, and it should become a regression test.