Safe Design for a Break-Glass Account After an Outage
A financial services firm uses CyberArk PAM with a two-person integrity model for its most sensitive accounts. During a weekend datacenter incident, the primary PAM vault and its DR replica were both unreachable for 90 minutes because a network change isolated the PAM subnet. The on-call platform lead needed the domain Administrator account to restore connectivity. The break-glass account existed in the vault, but the vault was down - so the lead used a locally cached credential from a password manager on their laptop. The incident was resolved, but the CISO is furious: the break-glass process 'didn't work when we needed it,' and the cached credential has now been used outside PAM with no session record. The team proposes storing the break-glass password in a sealed envelope in the datacenter safe and calling it done.
1. Assess why the break-glass process failed and whether the sealed-envelope proposal is adequate.
Read the worked answer
The failure was architectural, not procedural: the break-glass credential was stored in the same system whose outage it exists to survive. A break-glass mechanism that depends on the vault being available is not a break-glass mechanism. The 90-minute PAM-subnet isolation is exactly the scenario it must tolerate.
The sealed envelope in the datacenter safe is better than a laptop password manager - it's physically controlled and out-of-band - but it's not adequate on its own:
- No reliable record of who opened it and when (a logbook is only as good as the discipline around it).
- No rotation trigger - the password is static until someone remembers to change it.
- No way to tell whether it was used, or by whom, after the fact.
So the envelope isn't wrong as a last-resort physical layer; it's wrong as the whole answer. The judgment: match the recovery mechanism to the failure mode. If the vault can be unreachable, the recovery path must not live inside the vault.
2. Design a break-glass approach that survives a vault outage while staying auditable. What controls make it trustworthy?
Read the worked answer
Layer it so no single outage defeats it:
- Out-of-band storage, independent of the PAM network path. Options: a sealed physical credential in a controlled safe plus a geographically separate secondary vault/DR that is not on the same isolated subnet. The point is that the recovery secret's availability doesn't share fate with the primary vault.
- Tamper evidence + custody. Sealed envelope with serialized tamper tape, signed in/out by two people where feasible, stored where physical access is logged.
- Detective controls that fire on use. The break-glass account should be one whose any use generates a high-severity alert to security - because legitimate use is rare and expected to be announced. Wire authentication logs (e.g., domain controller events for that account) to alerting.
- Forced rotation after use. Any break-glass use - envelope opened or DR credential used - must trigger an immediate reset and re-seal, with a ticket.
- Periodic test. Break-glass that is never tested is break-glass that fails when needed (this incident proves it). Schedule a drill, including a vault-unreachable scenario, and record the result.
- Minimize the account's standing rights. Break-glass should be powerful enough to restore access, not a permanent domain-wide superuser with no monitoring.
The trust comes from detection and rotation, not from secrecy alone. A static password nobody watches is the weakest link; an out-of-band secret that screams when touched and is rotated after is defensible.