Network Observability and Telemetry: From Counters to Answers
Core concepts
Observability answers questions you did not predefine; monitoring answers questions you did. Candidates must distinguish the three pillars - metrics, logs, and traces - and know where network telemetry fits. Understand push versus pull models: SNMP polling is pull-based and coarse; streaming telemetry (gNMI, model-driven) is push-based, high-frequency, and structured. Know the telemetry types: counters and gauges, flow records (NetFlow/IPFIX with templates), and in-band or postcard-style telemetry that carries per-packet or per-hop metadata for precise latency and drop localization. Time synchronization (NTP/PTP) is essential - timestamps across devices are useless if clocks drift. Understand sampling trade-offs: full flow export is expensive, sampled export loses tail events. Observability pipelines need normalization, enrichment (device, site, tenant), and retention policies.
Traps
Questions confuse correlation with causation - a spike in drops may follow, not cause, a routing change. Another trap: assuming SNMP gives per-flow detail; it does not. Watch for clock skew invalidating cross-device latency analysis, and for sampled telemetry missing short microbursts. Also distinguish symptom metrics (interface errors) from root-cause signals (queue depth, buffer drops).
Worked example
An application shows intermittent latency. Steps: compare timestamps across hops (verify NTP), then use per-hop telemetry to find where delay is introduced, then check queue drops on that hop. Rule of thumb: localize before you optimize - never tune QoS globally until a single hop is identified as the source.
Rule of thumb
Ask what question the data can answer; if the resolution or sampling cannot see the event, add telemetry rather than thresholds.