Long-lived admin keys after a public leak
A company has 60 developers in four teams and one NCP account. Each team stores a long-lived access key in CI variables; the keys carry Administrator permissions because that was faster to set up. A public repository exposed one key for 40 minutes before removal. Cloud Activity Tracer shows the key listed buckets in three regions and downloaded 2 GB from a bucket containing customer invoices. The company has 12 Object Storage buckets, 40 Server instances, three Cloud DB for MySQL instances, and one VPC per environment. They must write an incident report, prevent recurrence, and keep developers productive. Security wants least privilege, short-lived credentials, and centralized audit logs. The CEO asks whether a new account is required. The current on-call rotation has no security engineer, and the next audit is in 30 days.
1. What are the first five actions after the leak is confirmed, and why is deleting the key not the first step?
Read the worked answer
- First, disable the access key in Sub Account. Disabling stops API use immediately while preserving the key identity for investigation. Deleting it first can destroy the link between the key and the activity trail.
- Second, revoke active sessions and rotate every key that shares the same CI secret store. Assume the repository had more than one secret; the exposed key is not necessarily the only one copied.
- Third, preserve and export Cloud Activity Tracer logs for the 40-minute window and beyond. Identify source IPs, API calls, buckets listed, objects downloaded, and any new users, buckets, or policies created. Check for persistence, not just data access.
- Fourth, contain data exposure: identify the invoice objects, apply bucket policy changes if needed, block the source IP range at the network or bucket policy layer, and notify legal and customers according to the incident policy.
- Fifth, rotate credentials for the CI system itself and scan the repository history and build logs for other secrets. Deleting the key is a cleanup step, not containment; disabling and rotating come first.
2. Redesign identity for 60 developers and four teams. What belongs in Sub Account groups, policies, and CI, and how do you avoid repeating the admin-key pattern?
Read the worked answer
- Create Sub Account groups per team: web, data, platform, and analytics. Grant each group only the actions its team needs. For example, analytics gets read access to specific bucket prefixes, not all 12 buckets, and no permission to create users or change VPCs.
- Scope Object Storage policies by bucket and prefix. Customer invoice buckets should be readable only by the billing group and only from approved networks. Remove Administrator from every human group; keep a small break-glass group with approval and alerting.
- Separate human console access from workload access. Developers use their own Sub Account with MFA for the console. CI uses a dedicated service identity per environment, with a key rotation schedule and a policy limited to the deployment target. Do not put one shared key in every pipeline.
- Prefer short-lived credentials where NCP supports them. If long-lived keys remain, rotate them automatically, store them in a secret manager, and alert on use from unexpected IPs.
- A new account is not the first control. Fix least privilege and key lifecycle in the current account first. If the audit requires a separate billing or regulatory boundary, evaluate a separate account later, but do not use account separation to avoid fixing IAM.
3. What audit, encryption, and monitoring controls would you put in place before the 30-day audit?
Read the worked answer
- Centralize Cloud Activity Tracer logs into a dedicated Object Storage bucket with write-only access from the logging pipeline and restricted read access for auditors. Set retention longer than the audit window.
- Alarm on anomalous Object Storage downloads, for example a GetObject spike above the normal baseline, access from a new country, or bucket policy changes. Route high-severity alarms to the on-call rotation even though no security engineer is on it; define an escalation to the security lead.
- Encrypt data at rest with KMS-managed keys where the service supports it, and enforce TLS in transit. Restrict bucket access to private endpoints and approved VPCs. Do not rely on obscurity of bucket names.
- Use Security Monitoring where available to detect suspicious behavior, and review the results weekly. Add a post-incident review with owners and dates, not just a report.
- For the audit, produce evidence: key rotation records, group membership, policy diffs, the incident timeline, and proof that the leaked key is disabled. The CEO question about a new account should be answered with the control gap and the chosen boundary, not with an unplanned migration.