A shared account, a leaked key, and least privilege
A 40-person startup runs everything in one Alibaba Cloud account. All engineers use a shared RAM user whose AccessKey is stored in a wiki page and baked into CI scripts. Last week a former contractor's laptop was stolen; the laptop had the wiki cached. Billing shows a spike in ECS launches in an unused region. The CTO wants to rotate the key and move on.
1. Assess the CTO's plan. What has actually happened and what must be done beyond rotation?
Read the worked answer
Rotation alone is insufficient because the exposure is structural, not a single secret. A shared user means the key cannot be attributed to a person, so the team cannot tell who used it or whether the stolen laptop's copy is the one that launched the instances. The spike in an unused region indicates the key was used to create resources outside normal operations, which is consistent with abuse.
Immediate actions:
- Disable the shared user's AccessKey and any other credentials tied to it; rotate secrets in CI and the wiki, and purge the wiki history.
- Inventory and terminate the unauthorized ECS instances; check for other created resources (snapshots, EIPs, RAM roles) and for persistence such as new access keys or modified policies.
- Review action logs for the exposure window to scope what was accessed, not just what was created.
- Notify stakeholders per policy if data may have been touched.
Structural fixes: eliminate the shared identity, give each engineer and each CI pipeline its own RAM identity with least-privilege policies, store secrets in a managed secret store rather than a wiki, and restrict which regions and actions the account permits so a single key cannot spin up infrastructure in an unused region unnoticed.
2. How would you design the identity model so this cannot recur, and how do you verify it works?
Read the worked answer
Design:
- Separate human and machine identities. Engineers authenticate as individuals (SSO/federation where possible) with MFA; CI uses dedicated RAM roles assumed by the pipeline, not long-lived keys stored in scripts.
- Least privilege by role. Define roles such as deployer, read-only auditor, and admin, each with policies scoped to the resources and actions needed. Deny by default; grant broad permissions only through a break-glass role that is time-bound and logged.
- Guardrails. Restrict allowed regions and high-risk actions at the account or organization level so a compromised identity cannot create resources in an unused region.
- Secret hygiene. No credentials in wikis or repos; use a secret manager with rotation, and scan repos for leaked keys.
Verification:
- Attempt a denied action with each role and confirm it fails (negative testing).
- Confirm logs capture the identity, source, and action for every request, and alert on anomalous patterns such as new-region resource creation or policy changes.
- Run a periodic access review so unused roles and keys are removed. The test of the model is not that it looks tidy but that a stolen credential produces a detectable, bounded event rather than a silent account takeover.