I85PREP
⌘ K
Consul Ops Pro study materials
Consul Ops Pro study resources

Expired snapshot token and a scoped KV wipe: full restore or surgical import?

Consul Ops Pro · Put your knowledge to work in context.

All case studies

Expired snapshot token and a scoped KV wipe: full restore or surgical import?

Fabrikam runs a three-server Consul cluster with ACLs, Connect and 3,000 keys under config/. Nightly at 02:00 a systemd timer runs consul snapshot save using a token issued by Vault with a 24-hour TTL; the timer's exit code is not monitored. Nineteen days ago the Vault token expired, and every snapshot since has failed; the newest file in the bucket is 19 days old. Yesterday an operator debugging a feature flag ran consul kv delete -recurse config/ against production instead of staging. config/ now contains four keys, and dashboards backed by those flags are serving defaults. Service registrations, intentions and ACLs are current and healthy. The team has a 19-day-old snapshot and asks whether to restore it to the running cluster, arguing that Consul's snapshot restore is the fastest path and that service registrations will re-register themselves anyway.

1. Should they run `consul snapshot restore` against the production cluster? Reason about what the snapshot contains versus what has changed in 19 days.

Read the worked answer

No, not for this incident. A Consul snapshot is the full Raft state: KV, catalog and service registrations, intentions, ACL policies and tokens, Connect CA state, and prepared queries. Restoring it replaces everything with the state 19 days ago, not just config/.

Cost of a wholesale restore here:

  • 19 days of KV writes outside config/ are reverted, including any keys created after the snapshot.
  • ACL policies and tokens created in the last 19 days vanish; any automation holding those tokens breaks immediately.
  • Intention changes and new service registrations are rolled back. Agents will re-register services via anti-entropy, but registrations that were made through the catalog API (which the team also uses) will not come back.
  • Connect CA state is also from the snapshot; if the CA was rotated in the last 19 days, restoring can invalidate issued certificates.

What is actually broken is a single subtree. The snapshot is still extremely valuable, because it contains the previous contents of config/.

Preferred path: use the snapshot as a data source, not a cluster replacement. Restore it into a scratch single-node cluster, export the config/ subtree, review the diff against current expectations, and import only that subtree into production. The blast radius stays at four keys plus the missing subtree.

2. Write the surgical recovery procedure, and the full-cluster restore procedure in case a future incident really needs it.

Read the worked answer

Surgical recovery (use now).

  1. Stand up an isolated single-node Consul (separate data directory, bootstrap_expect = 1, no network path to production).
  2. consul snapshot restore -token=<management> snapshot.snap against it. The snapshot carries its own ACL state, so authenticate with a token that exists inside it.
  3. consul kv export config/ > config.json and inspect the output; diff key names and values against what applications expect, and check for keys added after the snapshot that must not be reintroduced with stale values.
  4. In production, consul kv import @config.json with a token holding key_prefix "config/" write. Import merges by key; it does not delete extraneous keys, so confirm the four surviving keys are intentionally overwritten.
  5. Destroy the scratch cluster.

Full restore (for a real cluster-wide loss).

  1. Stop all servers. Snapshot restore must target a single server with no leader.
  2. Start one server alone (bootstrap_expect = 1), with its data directory cleared if it holds old state.
  3. consul snapshot restore -token=<management> file.snap, then verify consul operator raft list-peers shows one voter and spot-check KV and ACLs.
  4. Start the other two with bootstrap_expect = 3; they receive state from the leader. Revert the temporary bootstrap setting and restart the first server.
  5. Verify three voters, one leader, autopilot health, and that clients have re-registered services. Document everything lost since the snapshot.

3. Fix the backup pipeline so this is detected in hours, not weeks. What specifically must change?

Read the worked answer
  • Stop using a short-TTL token with no renewal. Either run a Vault Agent sidecar that renews the token, or use a dedicated long-lived least-privilege token scoped to the snapshot endpoint's required privileges, stored and audited in Vault. A 24-hour token on a nightly job is a scheduled outage.
  • Alert on snapshot age, not job execution. Publish a timestamp metric after each successful save (textfile collector or push gateway) and page when now - last_success > 26h. Also alert on non-zero exit codes from the timer; several failed nights in a row should never be silent.
  • Validate every artifact. Run consul snapshot inspect after each save and fail the job on a malformed or unexpectedly small file. Optionally restore into a scratch environment weekly as a full rehearsal.
  • Protect the artifacts. Ship to object storage with versioning and object lock, separate credentials, and a retention window longer than the worst realistic detection time. The 19-day gap is only survivable because the bucket still had older copies.
  • Reduce the blast radius on the human side. Remove recursive delete permissions on config/ from interactive tokens; require a change-management token or a two-person approval for kv delete -recurse. The incident was caused by a typo, and backups were the second failure, not the first.