Monitoring, Logging and Runtime Security
Runtime security is about detecting and responding to what containers actually do. The two exam pillars are audit logging and runtime threat detection.
For audit logs, configure the kube-apiserver with --audit-policy-file and --audit-log-path plus rotation flags, and write an audit policy whose rules are evaluated in order - the first matching rule wins. Levels are None, Metadata, Request, and RequestResponse; use omitStages: [RequestReceived] to cut noise. Rules match on users, groups, verbs, resources, namespaces, and non-resource URLs.
For runtime detection, know Falco: it consumes kernel events through a driver and evaluates rules, alerting on things like a shell spawned inside a container, writes to /etc, or unexpected outbound connections. Understand immutable container patterns such as read-only root filesystems, and that investigating a compromised pod means examining process trees, unexpected binaries, and anomalous network flows.
Traps: placing a catch-all rule before specific rules, since order decides the match; forgetting that audit logging is enabled on the API server, not a separate controller; and assuming Falco blocks activity - by default it only alerts.
Worked example: a policy rule with level: Metadata, resources pods, verbs create and delete, placed before a catch-all rule at level: None, logs pod lifecycle changes without recording request bodies.