I85PREP
⌘ K
Cilium Certified Associate study materials
Cilium Certified Associate study resources

Cilium Certified Associate: Observability and Troubleshooting

Cilium Certified Associate · Work through your exam, one topic at a time.

All guide sections

Observability and Troubleshooting

Observability and Troubleshooting tests whether you can determine why a packet vanished: which verdict to look for, which component to query, and where the agent keeps state.

Hubble: flow-level truth

Hubble is fed by the agent's eBPF datapath and exports per-flow records with source and destination identities, ports, verdict (FORWARDED, DROPPED, ERROR, AUDIT) and drop reason. Each agent embeds a Hubble server; the relay aggregates across nodes and the UI fronts the relay. Without the relay, cross-node queries and CLI calls to the relay service fail even though individual agents still collect flows.

The diagnostic ladder

  1. cilium status - agent health, kube-proxy replacement state, IPAM, Hubble. Use --verbose for per-node detail.
  2. cilium endpoint list - identity plus per-direction policy enforcement mode (Disabled, Audit, Enabled). A direction still Disabled means no policy selects it, so an empty allow list is not the problem.
  3. hubble observe --pod X --verdict DROPPED - surfaces the drop reason, separating policy denial from No route, TTL exceeded or conntrack conflicts.
  4. cilium monitor --type drop or --type policy-verdict - live datapath view for one node.
  5. cilium connectivity test - provisions its own namespace and exercises pod-to-pod, pod-to-service, DNS and external egress.
  6. cilium sysdump - bundles logs, map state and configuration for escalation.

Worked example

A client pod times out calling a service. hubble observe --pod client-xxx shows the SYN leaving and no reply, so the return path is suspect. cilium endpoint list shows the backend's ingress enforcement as Enabled, and cilium policy get reveals the policy selects app=api but only allows peers labelled app=frontend, while the client carries app=web. Fix the label or the selector. Had the verdict shown DROPPED with a policy reason, you would have skipped packet capture entirely.

Traps

Running cilium monitor unfiltered and drowning in noise; expecting L7 visibility without a visibility annotation or L7 rule; assuming hubble observe shows ClusterIPs for same-node pod-to-service flows, where socket-level load balancing already resolved the backend; and treating every DROPPED verdict as a policy denial.