Telemetry Stream Overwhelms Collector After Adding Interfaces
A team deploys gNMI telemetry from 24 Junos devices to a single collector. Initially they subscribe to /interfaces/interface/state/counters at 10s intervals on 8 uplinks per device. After a change window they add all interfaces: 48 per device. The collector's CPU pins at 100%, the Kafka topic backs up, and dashboards show stale data. The team's options discussed: (a) drop to 60s intervals, (b) filter to only interfaces with oper-status == up, (c) add a second collector and shard by device, (d) switch from STREAM/SAMPLE to STREAM/ON_CHANGE. The team also has an alert rule that fires when in-octets stops increasing for 5 minutes.
1. Estimate the change in message volume and rank the four options by effectiveness for the stated symptoms.
Read the worked answer
Volume scales with (devices × interfaces × paths) / interval. Before: 24 × 8 = 192 series at 10s → ~19.2 msg/s per path. After: 24 × 48 = 1152 series → ~115 msg/s per path, a 6× increase (interfaces went 8→48). If each sample carries multiple counters, multiply accordingly.
Ranking for the symptoms (collector CPU, Kafka backlog, stale dashboards):
- (b) Filter to
oper-status == up- highest leverage if many interfaces are down/unused. Directly cuts series count. Requires the device to support path/key filtering. - (d) ON_CHANGE - big win for counters that change slowly or not at all (down interfaces), but counters like octets change constantly on active links, so gains vary. Good complement to (b).
- (a) 60s interval - 6× reduction, simple, but reduces resolution; the 5-minute alert still works at 60s (5 samples).
- (c) Second collector - scales capacity but does not reduce volume; also doubles operational surface. Best as a last resort or combined with sharding.
Best combination: (b) + (d), with (a) as a fallback for high-churn paths, and (c) only if volume remains high after filtering.
2. The 5-minute 'in-octets not increasing' alert now fires spuriously on some links. Explain why and how to fix it.
Read the worked answer
Two likely causes:
- ON_CHANGE on a link that is genuinely idle or down: if the interface is down,
in-octetsdoes not change, so no samples arrive - indistinguishable from a stalled link. The alert must be conditioned onoper-status == upand on the presence of recent samples. - Stale data from a backed-up collector: if the pipeline lags, the alert evaluates old samples and sees no increase. The alert should evaluate sample timestamps, not wall-clock arrival, and should suppress when the pipeline lag exceeds a threshold.
Fix: change the rule to (a) require oper-status == up, (b) compare the latest two samples by their embedded timestamps, (c) add a 'no samples received' separate alert with its own threshold, and (d) add a pipeline-lag guard. This separates 'link is dead' from 'telemetry is dead' - conflating them is the classic mistake.
3. What should the team measure to decide whether a second collector is actually needed?
Read the worked answer
Measure, per collector: sustained CPU and memory, ingest msg/s, Kafka producer lag, and end-to-end sample latency (device timestamp → dashboard). Also measure per-device series count and the actual down/up interface ratio - that tells you how much filtering would save. A second collector is justified when, after filtering and interval tuning, sustained CPU stays above ~70% at peak or Kafka lag grows monotonically. Shard by device (consistent hashing) so a collector failure only affects a subset, and make the sharding key stable. Without these numbers, adding collectors is guesswork.