SCIM Deletion, a Shared Vault, and a Three-Year-Old Admin Key

Requirements, in order of what actually failed:

• Reachability. The runbook must be available without SSO: printed copy in a sealed envelope in the on-call safe, plus a copy in the on-call tooling that authenticates locally. The outage proved that documentation behind the broken dependency is not documentation.
• Independence. Break-glass identities must not be federated, must not depend on SCIM group membership, and must not be in the same blast radius as the IdP. IAM users in a dedicated security account are acceptable; permissions should be narrow but sufficient: IAM, Identity Center administration, and the ability to restore group assignments - not AdministratorAccess on everything.
• Non-shared custody. One credential per named responder, or sealed credentials with a two-person retrieval rule and a checkout log. A vault readable by the whole team makes attribution impossible and turns every team member into a permanent privilege holder.
• Phishing-resistant MFA. FIDO2 hardware keys stored physically separate from the credentials. Virtual MFA seeds in the same vault as the password are a single compromise away from being useless.
• Detection. EventBridge rule on ConsoleLogin by the break-glass user names routed to the CISO and the on-call pager. Any use is an incident until proven otherwise.

Proof: quarterly test where a responder actually authenticates, performs the restore action (re-creating a group and confirming SCIM reprovisioning), rotates the password, updates the runbook, and files the evidence - ticket, CloudTrail excerpt, and time-to-restore. A test that stops at 'login works' does not prove recovery. Review every use within 24 hours and rotate after any team departure.

Reject it, but separate the emergency from the permanent fix.

• Direct user assignments bypass the group-based lifecycle that SCIM manages. When a joiner or leaver is processed, direct assignments are invisible to the IdP and become orphaned access - precisely the finding the audit is about to look for.
• Making 260 engineers standing administrators converts a two-hour outage into a permanent privileged population. The next credential compromise now has 260 candidate entry points into full control of 41 accounts.
• The proposal does not address the actual root cause. The engineers were locked out because group names were recycled in the IdP, and because there was no monitoring of SCIM drift or identity-source changes.

What to do instead for the outage: restore the IdP groups with their original names and memberships; SCIM will reprovision the assignments. Where that is slow, use time-bound Identity Center assignments to a temporary elevated permission set with an expiry, tracked in the incident, and removed when the incident closes.

What to do so it does not recur: a change freeze or peer review on identity-source group renames, an alert on group deletion and on SCIM provisioning errors, a canary account that the platform team logs into through SSO every 15 minutes, and a documented recovery time objective for federation that has been tested. Capability restored by widening privilege is not resilience; it is a second incident waiting for a trigger.

Ranking rationale: the pipeline key is long-lived, unrotated, unrestricted, and holds AdministratorAccess - any leak is an immediate full-account compromise with no expiry to race. The break-glass weakness is lower frequency but higher impact. Both fit in eight weeks; the key fits in two.

Immediate (week 1–2):

• Rotate the key today and confirm no consumer breaks; then attach an inline policy with aws:SourceVpce or aws:SourceIp conditions limited to the CI egress or VPC endpoint as a compensating restriction while migration proceeds.
• Alert on any use of that key's ID outside the pipeline window; the identity is known, so baseline deviation is easy to detect.

Target state (week 3–6):

• Replace the static key with OIDC federation: the CI provider issues a short-lived token, AWS trusts the provider, and the job assumes a role. The trust policy must pin the repository, branch, and environment (sub claim), otherwise any repository in the organization can assume it.
• Drop AdministratorAccess. Deployment roles get the specific actions they need, scoped to the target accounts, with a permissions boundary to cap what the role can ever do and a separate, manually approved role for production.
• Separate roles per environment so a staging compromise cannot touch production.

Verification and evidence: CloudTrail showing the OIDC AssumeRoleWithWebIdentity events and zero use of the old key ID after cutover, the trust policy in version control, and the key deleted (not just deactivated) after a 30-day observation window. Report the remaining access-key inventory, per human and per machine identity, to the audit.