Remediating an audit on sub accounts, API keys, and encryption keys

Start from the blast radius. The main account key in Git is the single worst finding: it is the tenancy owner, it can create and delete anything, and it may carry billing authority. Assume it is compromised - rotate it, then verify what it was used for in the audit and access logs. Rotation is not optional and not something to schedule after the other four items.

Target model.

• Humans get individual sub accounts, grouped by role (app-operator, db-operator, network-admin, auditor), with console access behind MFA. No shared logins, ever.
• Machines get their own sub accounts, one per pipeline and per environment (ci-prod-deploy, ci-prod-registry), with no console access. Never one shared CI identity across environments, or a staging compromise deploys to production.
• Policies are narrowed per service and per need. Manager on every service is the root cause of finding (2): it was granted to make a deployment stop failing, which means the actual missing permission was never identified and thirty other permissions are now silently open. Fix the underlying permission gap, then shrink the policy.
• Compensating controls matter because platform policies are coarse: use IP allowlists on machine accounts (CI runner egress only), separate VPCs per environment so a mistaken policy still cannot reach production data, and Cloud Log Analytics on API activity.

Safe sequence. Create the new least-privilege identities → generate their keys and store them in the pipeline's secret store, not the repo → run the pipeline in parallel with the old key and confirm success → switch and observe for a full deploy cycle → disable the old credentials → rotate the main account key last, having proven nothing depends on it. Broad policies are shrunk one service at a time, with the audit trail showing which requests were denied and whether they were legitimate.

Environment variables in a Deployment are not secret. Anyone with read access to the namespace, and anyone who can run kubectl describe pod or read the ReplicaSet spec, can see them; they also leak into kubectl get pod -o yaml output copied into tickets and archives. Moving the password to a Kubernetes Secret is a marginal improvement - Secrets are base64, not encryption, and the same read permission exposes them.

Use the managed secret store. Put the credential in Secret Manager, give the workload's identity read-only access to that one secret, and have the pod fetch it at startup (or via an injected volume) rather than having it baked into the manifest or the image. Benefits beyond hiding the string: one place to rotate, an access log per read, and revocation without rebuilding images. If the app cannot be changed quickly, at minimum stop putting the value in the Deployment spec and rotate the credential - it must be considered disclosed.

KMS key separation is about the two-of-two rule. Today the same administrator group can use the key and delete it. Deleting a key is destructive beyond recovery: data encrypted with it becomes permanently unreadable even if the bucket still exists. Split the roles so that key administrators manage lifecycle and key users can encrypt/decrypt, and neither group has both. Keep key policies under a security owner outside the application admin group, enable deletion protection or a long scheduled-deletion window, and verify that Object Storage server-side encryption and the Cloud DB instances use keys with the right scope rather than one shared key for everything.

What this buys. Containment: one stolen credential or one mistaken console action can no longer decrypt or destroy the settlement archive, and you get an audit record of who touched the key.

Contractors: scope, time, and evidence. Create three individual sub accounts - never a shared one, because shared accounts make the audit log worthless. Grant a read-only policy limited to the log service and only the log projects in scope; do not grant server access, because reading logs does not require it. Restrict console access by IP allowlist to the contractor's office/VPN, require MFA, and issue no API keys unless automation genuinely needs them. Because platform sub accounts do not reliably self-expire, enforce the end date mechanically: a scheduled job or ticket that disables the accounts and revokes keys at day 21, with an owner named, plus a review at day 14 to decide whether an extension is justified. Log access during the period and compare it against the stated purpose at closure.

SSH exposure. The ACG rule allowing 22/tcp from anywhere is the reason you are seeing brute-force attempts; the fix is not better passwords. Sequence it so people are not blocked:

1. Stand up a bastion path first - a single hardened jump host with its ACG restricted to the office range, or a VPN into the VPC - and confirm every engineer can reach every server through it.
2. Keep key-based authentication only; disable password authentication.
3. Then remove the 0.0.0.0/0 rule from the server ACGs so servers accept 22 only from the bastion or VPN range.
4. For routine access, prefer a session-recording or audited access path, and use Security Monitoring and IPS to block repeat offenders during the transition.

Tie it together. The SSH attempts and the audit findings are the same problem: broad standing access. Fixing the contractors' scope and the SSH rule together reduces both the attack surface and the number of identities that need reviewing next quarter.