Audit disk full at 03:00 on a three-node auto-unseal cluster

Vault fails closed on auditing. If no audit device can write, Vault refuses to process the request rather than serve it unaudited - that is deliberate, and it is why reads of an unrelated KV path returned 500. Nothing was wrong with those requests; the cluster simply would not perform work it could not record, and the error text names the audit failure rather than the secret path, which is what sent on-call looking in the wrong place.

The restart could not help. Audit device configuration is stored in the barrier, not in the config file, so the node came back with the same device pointing at the same full volume. The only effects were a leadership change and a few seconds of extra unavailability. Restarting is the right reflex for a hung process; it is the wrong reflex for a failing audit backend.

Correct immediate action: restore writability of the target - clear or move space on the volume, or repoint file_path to a device with room - and the device resumes. File audit devices support re-opening on SIGHUP, so a rotate-then-SIGHUP cycle is a safe way to free the file without losing configuration.

The lesson to write into the runbook: the second audit device has to be in place before the incident, because the window in which you need to add one is exactly the window in which the API is refusing requests.

It covers data loss, not key loss. The snapshot protects against corrupted or deleted secrets, a botched upgrade, a bad policy purge, or a node's disk dying - all recoverable by restoring into a cluster.

The gap: the KMS key. With auto-unseal, the barrier keyring is wrapped by the cloud KMS key. The snapshot contains barrier-encrypted data whose keyring entry is unwrapped only by that KMS key. Delete or lose access to the KMS key - a console mistake, an over-broad IAM change, a deleted key version - and every snapshot becomes an expensive encrypted brick. Recovery keys do not help: they authorise root token generation and rekey, they do not unseal a cluster that uses auto-unseal. The team has three people holding shares of a mechanism that does not cover their actual single point of failure.

Close the gap:

• Deletion protection and a restrictive key policy on the KMS key; separate the key administrator from the Vault operator role.
• Replicate or back up the key material to an account the Vault team cannot unilaterally destroy.
• Quarterly restore drills into an isolated cluster. A snapshot that has never been restored is a hypothesis, not a backup - and restore is where mismatched seal configuration shows up.

Also treat a leaked snapshot as sensitive: it is encrypted, but it is still the whole cluster.

mlock prevents Vault's memory from being paged to swap. With it disabled, the unsealed master key material and plaintext secrets can in principle be written to swap or a core dump - a disk-resident copy of things the whole design exists to keep only in memory. The exposure is real but conditional: it requires the host to swap or to capture a core file, and an attacker who can read those is already deep in the host.

Proportional response, cheapest first:

1. No swap on Vault nodes (or encrypted swap). This removes most of the risk without changing the image.
2. Disable core dumps for the container and restrict host access to the Vault nodes - the operator who can read a core file is the threat.
3. Treat disable_mlock=true as a tracked exception with an owner and a date, and make IPC_LOCK a requirement for the next base image. Containers can receive the capability; the blocker is image policy, not a technical impossibility.

Do not silently accept it: the sentence "we disabled mlock" in a config file with no ticket is how this becomes permanent. And verify the setting took effect the way you expect - if IPC_LOCK is later granted while disable_mlock=true remains, you get the maintenance behaviour of one without the protection of the other.