Upgrades, Maintenance, and Operational Runbooks
What this domain covers
Upgrades, Maintenance, and Operational Runbooks tests ordered procedures: rolling upgrades, feature compatibility versions, index builds, and routine maintenance.
Core rules
- Upgrade binaries before raising the feature compatibility version (FCV). Use setFeatureCompatibilityVersion only after every member runs the new binary.
- Replica set rolling upgrade: upgrade secondaries one at a time, step down the primary, upgrade it, then raise FCV.
- Sharded cluster upgrade: stop the balancer, upgrade mongos routers, upgrade the config server replica set, upgrade shards one at a time, then raise FCV.
- Index builds on a replica set can run on secondaries first, then the primary, to reduce impact.
- Maintenance includes log rotation with logRotate, validate for corruption checks, and compact for space reclamation. repair is offline and last-resort.
Worked example
Upgrade a three-member 4.4 replica set to 5.0. Back up first. Upgrade secondary B and C binaries, restarting each and waiting for SECONDARY state. Step down the primary, upgrade its binary, and let it rejoin. Confirm all members report 5.0, then run setFeatureCompatibilityVersion 5.0. Do not set FCV before the last binary is upgraded, or mixed-version features can fail.
Traps
Raising FCV too early; upgrading the primary first; forgetting the balancer in a sharded upgrade; running repair on a live node; and skipping a backup before maintenance.