The Break-Glass Account Nobody Watched
A payments processor maintains 14 privileged accounts in a cyberArk vault. Policy requires checkout approval for production database access. Last quarter, an on-call engineer used a break-glass account during a 2 a.m. outage because the approver was unreachable. The break-glass account is excluded from the normal approval workflow by design, but it is supposed to page the security on-call and open a ticket automatically. Neither happened: the paging integration had been disabled during a migration six months earlier and never re-enabled, and the ticketing webhook was pointing at a decommissioned endpoint. The engineer used the account for 40 minutes, ran three diagnostic queries, and closed it out. No one noticed until a routine log review found the session. The engineer's actions were legitimate and helpful. The audit committee asks how this could happen and what to change.
1. Explain the control failure. Was the break-glass design flawed, or the implementation?
Read the worked answer
The design was reasonable; the implementation and monitoring failed.
Break-glass exists precisely for the case where normal approval is unavailable-so excluding it from the approval workflow is correct, provided it is compensated by detection and after-the-fact review. The compensating controls were: (1) page security on-call, and (2) auto-open a ticket. Both were silently broken for six months. That is the failure: compensating controls were treated as 'set and forget' with no health monitoring.
The deeper issue is that a break-glass path with no working alert is indistinguishable from an unmonitored backdoor. The engineer did nothing wrong; the organization failed to notice that its safety net had holes. The lesson is that break-glass controls need their own monitoring-an alerting path that is tested on a schedule and whose failure itself generates an alert.
2. Recommend concrete changes, distinguishing quick fixes from structural improvements.
Read the worked answer
Quick fixes (this sprint):
- Re-enable the paging integration and repoint the ticketing webhook; test end-to-end with a synthetic break-glass event.
- Add a heartbeat/canary check: if the alerting path fails a weekly test, page the security team.
- Review all vault integrations for similar 'disabled during migration, never restored' drift.
Structural improvements:
- Session recording and automatic post-use review: every break-glass session triggers a mandatory review within 24 hours, assigned to someone other than the user.
- Time-boxed credentials: break-glass access auto-expires (e.g., 60 minutes) and requires re-justification to extend.
- Two-person rule where feasible: even break-glass requires a second party to acknowledge, with a documented fallback if truly unreachable.
- Control-health dashboard: compensating controls reported alongside primary controls so a broken safety net is visible.
- Post-incident blameless review that captures the legitimate use as a case study, reinforcing that using break-glass correctly is encouraged-but visible.
The framing for the audit committee: the engineer followed the spirit of the policy; the organization must make the safety net as reliable as the primary control.