Cortex Cloud onboarding: nine dark accounts and a duplicated AWS account

Hand-patching is the wrong primary path. Those roles are CloudFormation-managed: console edits create drift, and the next stack update will either fail on a resource conflict or silently revert the edit. Editing the trust policy in place also breaks the link between the role and the template version that produced it, so nobody can later prove what permissions exist.

Separate the two defects:

• SCP denial (3 accounts) is owned by the cloud platform team. Request a time-bound, condition-scoped exception (role ARN plus region condition) with a ticket reference, rather than a blanket allow.
• Missing external ID (6 accounts) is fixed by deleting the malformed role and re-running the current onboarding template so the stack owns the resource again. Do not edit in place.

Sequence: canary one non-production account, then verify the reported resource count against AWS Resource Explorer or Config for that account and region (accept a small delta for resources discovered later). Only then batch the remainder.

Acceptance criteria for the board date: all nine accounts report inventory, permission validation passes, alert flow resumes, and no orphaned duplicate role remains. If the SCP exception cannot be granted in time, onboard those three accounts with a role name that the SCP already permits and document the deviation.

A cloud account record is keyed on the cloud account identifier plus its credential set. Onboarding the same AWS account twice creates two records that both call the same Describe/Config APIs, so the resource inventory is stored twice and every config policy is evaluated twice. Alerts carry the resource, the policy and the originating cloud account, so the SOC sees two alerts for one bucket or one security group. Nothing is wrong with detection; the platform is doing exactly what it was told.

Cleanup sequence:

1. Use an RQL query grouping resources by cloud account to confirm which record is complete (permissions, regions enabled, resource count).
2. Identify whether either record carries scoped RBAC, alert rules, notification integrations or policy exceptions. Migrate those before deletion.
3. Delete the redundant cloud account record in the console first. Do not delete the IAM role in AWS until you have confirmed which stack owns it; otherwise you break the surviving record on its next credential refresh.
4. Confirm alert volume for the affected policies halves and that no account-scoped exceptions were lost.
5. Record the surviving record's owning stack so future onboarding does not re-create the duplicate.

Prevention should lean on the platform's own surfaces rather than a new approval committee.

• Onboard once, at the organization level. Make organization/tenant-level onboarding the only sanctioned path, and grant the cloud account create permission only to a single platform role. Application teams get read and exception rights, not onboarding rights. That alone eliminates the manual duplicate.
• A coverage canary query. Save an RQL query that counts resources per cloud account and schedule a daily alert if any account drops below a threshold or reports zero for more than 4 hours. Zero-resource accounts are the earliest symptom of both a broken trust policy and a deleted role.
• Permission validation as a gate, run in the pipeline that applies the onboarding template, so an SCP change fails a check rather than silently blinding the platform.
• Label ownership. Tag the onboarding stack and the owning team; require that tag in the account inventory view so a stray record is obvious.

Keep it lightweight: one permission change, one scheduled query, one pipeline check. The expensive part of this incident was 72 hours of undetected blindness, not the missing paperwork.