ServiceNow CIS-SecOps study materials
ServiceNow CIS-SecOps study resources

The Flow That Fails Silently and the MTTR That Fell

ServiceNow CIS-SecOps · Put your knowledge to work in context.

All case studies

The Flow That Fails Silently and the MTTR That Fell

Litware's SIEM connector pushes detection events into Security Incident Response. An endpoint rule and a network rule both fire on the same laptop within minutes, so each compromised device produces two or three incidents. A Flow Designer flow triggers on incident creation: it enriches the incident from threat intelligence, creates four response tasks, posts to a Teams channel, and updates the incident record with the enrichment result. The flow runs as the integration user, which lacks write access to the assignment group field on incidents created by one of the two detection sources; the failing action continues and the remainder of the flow proceeds. A dashboard built on incident close dates shows MTTR falling from 14 days to 3 days over the quarter, while analysts insist nothing has actually got faster.

1. Design a deduplication and correlation approach for the duplicate SIEM events. What are the trade-offs between doing it at the SIEM and doing it in ServiceNow?

Read the worked answer

You are dealing with two different things: duplicates (same event reported twice) and correlation (two genuinely different detections describing one intrusion). Handle them differently.

Duplicates - fix at the source where possible. If the same host and user generate the same detection from two rules, ask the SIEM owners for a correlation rule that emits one alert per entity per window, forwarding the second detection as an attribute. Cheapest, and it keeps ServiceNow's record count honest. Cost: you lose the second rule's independent evidence unless the attribute is preserved.

Correlation - keep in ServiceNow. Endpoint malware plus an outbound connection to a known-bad address are two facts that mean more together. Group them: a parent incident with children, or one incident with the second detection attached as evidence. Do not silently merge two different detections into one record with one description.

If you dedupe in ServiceNow, build the key deliberately: CI or host identifier, plus user, plus detection signature, plus a time bucket. Trade-off: too wide a window swallows a second, unrelated compromise on the same host; too narrow and you are back to three incidents per laptop. Test the window against a week of real data before enabling - and log what was suppressed, so a wrong merge is discoverable.

The failure mode to avoid is dedupe that only closes the duplicate: the analyst then works a record whose evidence sits in a closed sibling and cannot see it.

2. Audit the flow design. What are the highest-severity defects, and how would you fix each?

Read the worked answer

Defect 1 - silent continuation on permission failure. This is the worst defect in the list. An action that cannot write returns nothing useful, the flow continues, and the incident looks enriched when it is not. Fix: run the flow as a dedicated service account with an explicit, minimal role set scoped to the objects it touches - never the integration user, whose permissions are shaped by ingestion needs. Then make failure visible: on error, stamp the incident with an 'enrichment incomplete' flag or state and route it to an exception queue. A record that looks processed and is not is worse than no flow.

Defect 2 - non-idempotent actions. If the flow retries or fires twice - and record-based triggers frequently do - you get eight response tasks instead of four. Fix: check for an existing enrichment or existing tasks before creating, and key the created records to the flow context.

Defect 3 - recursion risk. The flow updates the incident it was triggered by. If the trigger condition is not restricted to creation, or is watching updated fields, the update can re-trigger the flow. Fix: trigger on created only, or gate with a condition that excludes updates made by the flow's own service account.

Defect 4 - no error telemetry. Fix: log action failures to a reportable record and alert on the count. Without it, the only signal that the flow is broken is an analyst noticing missing data - which, given Defect 1, they will not.

3. Analysts say nothing got faster, but the dashboard says MTTR fell from 14 days to 3. How do you determine which is true, and what would you publish instead?

Read the worked answer

Both can be true simultaneously, because they are probably measuring different populations.

Diagnose before you argue. Recompute MTTR from raw timestamps for a matched cohort: same category, same severity, one month before the flow and one month after. If the recomputed figure also drops, look for a distribution shift rather than a mean shift - a chunk of incidents now closing in minutes will drag a mean down without any real investigation getting faster.

Likely causes, in order:

  • The flow creates response tasks that a scheduled job or a closing rule auto-closes, changing the close path rather than the work.
  • The measurement's start point moved - if the flow's update stamps a field the report uses as the start, elapsed time shrinks artificially.
  • Incidents arriving from one detection source fail to get an assignment group (Defect 1) and are swept up by a cleanup job, so the slowest records never appear in the closed population. This one is the trap: the flow's bug and the metric's improvement are the same event.

Publish instead: median and P90 time from acknowledged to contained, split by severity; time spent waiting on approval or third-party response reported separately so the number reflects work rather than queue; and a data quality metric - percentage of incidents with complete enrichment and a valid assignment group. That last one would have caught this before the quarter ended.