Ceph Pool Becomes Inactive After a Second Host Reboot
Five Ceph Reef hosts, c1 through c5, each run 12 OSDs and one MON. Pool rbd uses replicated size 3, min_size 2, and CRUSH failure domain host. Host c3 failed four hours ago and was marked out; backfill is only 60 percent complete. The team begins a kernel update on c4 and reboots it without setting any Ceph flags. Within two minutes, ceph health detail shows PG_AVAILABILITY and many PGs in active+undersized+degraded; some are inactive. rbd clients report input/output errors. The on-call engineer proposes ceph osd pool set rbd min_size 1 to restore writes. The cluster still has three hosts up, and the affected PGs have one surviving copy on c5 or c1.
1. Evaluate the proposal to set min_size to 1. What does it actually change, and what risk does the team accept?
Read the worked answer
Lowering min_size to 1 can make the pool activate with a single surviving copy, but it does not create redundancy. Any failure of that last OSD, a bad sector, or a scrub error becomes permanent data loss. It also weakens the guarantee that writes are on two hosts. Before considering it, confirm the affected PGs really have one copy and no other copy is recoverable; check ceph pg dump and ceph osd map. If the business accepts the risk, use it only as a short, logged emergency change, keep noout and nobackfill from adding load, and revert to min_size 2 as soon as two copies exist. Prefer restoring c3 or c4 or adding capacity; do not make min_size 1 the normal setting.
2. Give a safe recovery sequence that does not make the outage worse, including Ceph flags and the order of host recovery.
Read the worked answer
Stop further maintenance, confirm c4 is coming back, and set ceph osd set noout so additional transient OSDs are not marked out. If c3 is still unavailable, bring it back or replace it; do not mark more hosts out. Use ceph osd set nobackfill only if recovery load is hurting clients, but remember it delays restoration of redundancy. Watch ceph -s, ceph health detail, ceph pg dump, and ceph osd df. Let recovery reach clean or at least all PGs active with two copies before clearing flags. If c4 cannot return promptly and only one copy remains, the least-bad choice may be min_size 1 with continuous monitoring, but that is an explicit data-loss risk, not a fix. Restore c4 first because it is the most recent planned change.
3. The pool is healthy again. What capacity and change-control practices would prevent a second host from causing inactive PGs?
Read the worked answer
The immediate cause was two overlapping failures with incomplete recovery. A five-host size-3 pool can tolerate two host losses only while all PGs still have two surviving copies; during backfill that margin disappears. Maintain enough hosts or failure domains that one host can be out and one can be rebooted without collapsing a PG below min_size. Always set noout before planned reboots, perform one host at a time, and verify recovery progress before the next change. Add capacity headroom for backfill, monitor PG distribution and degraded counts, and alert on PGs with fewer than two active copies. Treat min_size changes as emergency-only, time-bounded, and auditable. Test the maintenance runbook so no one reboots a host while another planned outage is still recovering.