Security and Observability
What this domain tests
Security and Observability covers protecting the pipeline and cluster, and knowing what the system is doing: TLS, RBAC, secret handling, image supply chain, plus metrics, logs, traces, and alerting.
Core rules
- Encrypt in transit everywhere: Ingress TLS, API server TLS, and mTLS between services (a service mesh automates certificate rotation).
- Apply least privilege with RBAC. Roles and RoleBindings are namespaced; ClusterRoles are cluster-wide. Grant verbs on specific resources, never
*on*. - Keep secrets out of Git. Use an external manager (Vault, cloud secret stores) or sealed secrets, mount them as files where possible, and enable encryption at rest for the API server.
- Scan images in CI (Trivy, Grype), sign them (cosign), and deploy by digest.
- Observability rests on three signals: metrics (Prometheus scrapes
/metrics), structured logs written to stdout and collected by the platform, and distributed traces via OpenTelemetry. - Alert on symptoms users feel - error rate and latency against an SLO - not on every metric. Track the golden signals: latency, traffic, errors, saturation.
Worked example
Prometheus reports a target DOWN after a release. Reasoning: first confirm the Service still selects the Pods with kubectl get endpoints, then compare the ServiceMonitor's selector.matchLabels with the Service labels - a mismatch means no scrape config is generated. If the config exists, check the declared port: the new Pod exposes metrics on 9090 while the ServiceMonitor says 8080, or a NetworkPolicy now blocks the scrape. Fix the port or label and the target returns UP. If metrics arrive but are incomplete, suspect relabeling dropping labels.
Traps
- High-cardinality labels (user IDs, request IDs) that explode the time-series database.
- Binding a ClusterRole for convenience where a namespaced Role suffices.
- Assuming
kubectl describe secretprotects values -get secret -o yamlreveals the base64. - Logging tokens or passwords, which then persist in the log store.