Telemetry Pipeline Overload After Full-Flow Export
A media streaming company enabled full NetFlow export on all 300 routers and switches to improve observability. Within a week, the collector cluster (3 nodes, 16 cores each) is dropping 40 percent of flow records, dashboards lag by 15 minutes, and the network operations team has stopped trusting the data. The company also wants to add streaming telemetry for interface counters and BGP state. Budget allows either scaling the collector or redesigning the telemetry strategy, not both. The CTO wants mean time to detect (MTTD) for network incidents under 5 minutes.
1. Should the architect scale the collector or redesign the telemetry strategy? Justify with reasoning about data volume and MTTD.
Read the worked answer
The architect should redesign the strategy, because scaling the collector treats the symptom and the 5-minute MTTD goal is not achievable with the current approach.
Reasoning:
- Volume math. Full-flow export from 300 devices at high traffic rates can generate millions of flow records per minute. Three 16-core nodes are already dropping 40 percent, so the pipeline is saturated. Adding nodes helps linearly, but the underlying approach - exporting every flow from every device - is the wrong tool for MTTD.
- MTTD needs different data. Detecting incidents in under 5 minutes requires streaming telemetry (model-driven, push-based, sub-minute intervals) for interface counters, queue drops, and BGP state - the signals that precede outages. Full flow records are better for forensics and capacity analysis, not fast detection.
- Sampling and selective export. Enable sampled flow (e.g., 1:1000) on high-volume links and full flow only where needed (WAN edges, security boundaries). This cuts volume by orders of magnitude while preserving useful data.
- Tiered pipeline. Route streaming telemetry to a fast time-series store for alerting, and sampled flows to a separate analytics store for investigation. Separate the detection path from the forensics path so neither starves the other.
- Validate against the goal. After redesign, measure MTTD with synthetic incidents and confirm sub-5-minute detection.
The trap is choosing "scale the collector" because it is simpler. The exam tests whether you connect the observability goal (MTTD) to the right data type and volume strategy.
2. How should the architect phase the redesign to avoid losing observability during the transition, and what success metrics prove it worked?
Read the worked answer
Phasing must never leave a blind spot while the old pipeline is being replaced.
Phased plan:
- Baseline and inventory. Record current flow volume per device, collector drop rate, and dashboard lag. Identify the top talkers generating most flow records.
- Stand up streaming telemetry in parallel. Deploy model-driven telemetry for interface counters, queue drops, and BGP state to a new time-series store. Keep the existing flow pipeline running unchanged.
- Build fast-detection alerts on streaming data. Create threshold and anomaly alerts for the signals that precede incidents (interface errors, queue drops, BGP session flaps). Tune to avoid alert fatigue.
- Reduce flow volume selectively. Enable sampling on high-volume links first, verifying that forensic queries still return useful results. Keep full flow on WAN edges and security boundaries.
- Cut over dashboards. Move detection dashboards to streaming data, keep forensic dashboards on sampled flows, and decommission redundant full-flow exports only after the new path is proven.
Success metrics:
- Collector drop rate near zero on both paths.
- Dashboard freshness under 1 minute for detection views.
- MTTD under 5 minutes, proven with synthetic incident tests.
- Forensic query completeness - sampled flows still answer the top investigation questions.
- Alert precision - ratio of true incidents to alerts, tracked over several weeks.
The judgement tested: parallel operation during migration, and defining measurable success rather than declaring victory on deployment.