Operations and Maintenance
Snapshots
consul snapshot save backup.snap captures the Raft state - catalog, KV, ACLs, intentions, config entries - from a server. Restore with consul snapshot restore backup.snap pointed at a server's HTTP API. Restores are whole-cluster state replacements: the safest pattern is to restore into a fresh server, then rejoin other servers. Snapshots do not include the gossip key, TLS material, or agent configuration files, and a snapshot from a newer Consul version cannot be restored to an older one.
Autopilot
Autopilot manages server health and upgrade safety. Key settings: cleanup_dead_servers (default true, removes failed servers from Raft after last_contact_threshold), max_trailing_logs, server_stabilization_time, redundancy_zone_tag (spread servers across zones), and disable_upgrade_migration. Inspect with consul operator autopilot get-config and consul operator autopilot health; the health output flags a server that is not stable or is behind on logs.
Upgrades and monitoring
Upgrade servers before clients, and one server at a time - followers first, leader last - so quorum is never lost. Consul supports rolling upgrades one minor version at a time; skipping minor versions is unsupported. Watch convergence with consul members, consul operator raft list-peers, and consul info. Metrics (Raft apply lag, leader elections, Serf health, check failures, RPC errors) flow to StatsD/DogStatsD/Prometheus via telemetry blocks; consul monitor streams logs at a chosen level.
Traps
- Restoring a snapshot does not restore TLS certificates or the gossip key.
cleanup_dead_serverscan remove a merely partitioned server - a deliberate trade-off, not a bug.- Upgrading all servers at once breaks quorum even though each node "works".
Rule of thumb
Snapshot before every upgrade; upgrade one server at a time, followers first.