Removing a Baked-In API Key Without Breaking a Running Fleet

Dynamic groups let the instances authenticate as themselves, so nothing long-lived is stored on the host.

• Create a dynamic group in the tenancy with a matching rule selecting exactly the prod instances: instance.compartment.id = 'ocid1.compartment.oc1..prod', or a tag-based rule such as tag.project.role.value = 'claims-app' if the prod compartment also hosts unrelated instances. Prefer the tag-based rule when the compartment is not dedicated.
• Write policies in the compartment that owns the resources, not the one that owns the instances. Policies are evaluated against where the bucket lives:
  • allow dynamic-group prod-app-instances to read objects in compartment shared-data (list + get)
  • allow dynamic-group prod-app-instances to manage objects in compartment shared-data for the log uploads
  • add read buckets only if the application enumerates buckets
• Place the statement at the tenancy root only if it must span several compartments; otherwise scope it to shared-data to keep the blast radius small.
• The instances then use instance principal authentication through the CLI/SDK - no key file, no config file, no secret in the image.
• Roll out in batches and confirm a canary instance can both read the data bucket and write the log bucket before anything touches the old credential.

Order matters: removing the key first will break the fleet mid-deploy.

1. Create the dynamic group and the narrow policies.
2. Deploy the instance-principal version to a canary instance in prod; verify the read on the data bucket and the write to the log bucket using the instance's own identity.
3. Roll the remainder of the fleet. Leave the old credential in place but unused until the rollout is verified.
4. Disable the originating user rather than deleting it immediately - disabling is reversible, stops the credential working instantly, and tells you within minutes whether anything still depends on it.
5. Delete the API signing key, plus any auth tokens, customer secret keys and console password for that user; then delete or repurpose the account.
6. Rebuild the custom image with the key material removed. If the key stays in the image, every future instance re-introduces the leak.

Treat the key as compromised, not merely misplaced: it was readable from boot volumes and embedded in an image. Pull Audit logs for that user/API key over its whole lifetime, look for calls from unexpected source IPs, for IAM or bucket policy changes, and for resources created under that identity. Then add a preventive control: no credential material in images, and a policy review gate on principals with manage rights.

• It grants every principal in the tenancy full control of every resource in every compartment - including sandbox users and any federated identity mapped into the tenancy. A low-privilege account becomes a tenancy administrator. It also obscures every application's real permissions, so the effective policy set can no longer be reasoned about.
• OCI IAM is additive and has no explicit deny, so this cannot be neutralised by writing a narrower statement. It must be deleted; the deletion is the control.

Remediation:

1. Delete the statement (record who added it and when, from Audit).
2. Replace it with group-based statements scoped to a compartment and a resource family, e.g. allow group app-admins to manage instances in compartment prod and allow group data-scientists to use object-family in compartment sandbox.
3. If a proof of concept genuinely needs breadth, isolate it in its own compartment or tenancy and keep any-user for one narrowly defined resource type, with an expiry date recorded in the change ticket.
4. Re-review all root-level policies: every subject should be a group, dynamic group or service principal, and every scope should be a compartment rather than the tenancy unless there is a documented reason.
5. Check Audit to see whether the statement was ever exercised, and by whom.