I85PREP
⌘ K
OCI Developer Professional study materials
OCI Developer Professional study resources

Object Storage 401 After OKE Migration: Instance Principal vs Workload Identity

OCI Developer Professional · Put your knowledge to work in context.

All case studies

Object Storage 401 After OKE Migration: Instance Principal vs Workload Identity

invoice-svc is a Spring Boot deployment on OKE (Kubernetes 1.28, three node pools, twelve services) that reads PDFs from Object Storage bucket invoice-archive and writes rendered copies to invoice-rendered. On the previous cluster it authenticated with instance principals and worked. After migration to a new cluster, every Object Storage call fails with HTTP 401 NotAuthorizedOrNotFound, while other services on the same nodes continue to work. The nodes run in compartment prod-infra; the pods run in namespace billing and use service account invoice-sa, which carries no annotations. The existing dynamic group rule is instance.compartment.id = 'ocid1.compartment..prod-apps'. The team's proposed fix is to broaden the rule to Any {instance.compartment.id = 'prod-infra'}. Two other workloads on the same nodes must not gain access to invoice-archive.

1. Explain why the calls fail and why broadening the dynamic group rule is the wrong fix.

Read the worked answer

Why it fails now. The old cluster's nodes lived in prod-apps, so the instance-principal rule matched them and every pod on those nodes inherited the node's identity. After migration the nodes are in prod-infra; the rule no longer matches, so node-level principals get no policy grants and Object Storage answers 401. Note the second-order detail: the services that still work are the ones whose buckets are granted to principals that do match (for example node-pool agents or a different compartment's rule) - the failure is specific to this bucket's policy, not a cluster-wide outage. That is the signal that identity resolution, not networking, is the problem.

Why the proposed fix is wrong. An instance principal is the node's identity. Broadening the rule to prod-infra grants every pod scheduled on those nodes - including the two workloads that must stay out of invoice-archive - the node's permissions. That is a lateral-privilege expansion across twelve services, it cannot express "namespace billing may read this bucket", and it breaks again the next time a node pool moves or a compartment is reorganised. It also violates the stated constraint directly.

Correct direction. Move the workload to a pod-scoped identity (OKE Workload Identity) so the grant follows the service account, and keep instance principals only for node-level daemons such as the CNI or CSI plugin.

2. Describe the concrete configuration needed for the service-account-scoped identity, including least-privilege policy, and how you would verify it.

Read the worked answer

Configuration steps.

  1. Confirm the cluster has Workload Identity enabled (a cluster-level setting; a cluster created without it needs the enablement step before any pod can use it).
  2. Create a dynamic group whose matching rule targets service accounts, e.g. All {resource.type = 'serviceaccount', resource.compartment.id = 'ocid1.compartment..prod-apps'}. Note the compartment used here is the one that owns the cluster/namespace resources, not the node compartment - mixing these up is the classic repeat of the original bug.
  3. Bind the Kubernetes service account billing/invoice-sa to that principal (the workload identity mechanism associates the namespace/service-account pair with the OCI principal; the projected token is mounted into the pod by the cluster's webhook).
  4. Reference invoice-sa in the Deployment's serviceAccountName, and let the OCI SDK pick up the workload identity provider instead of InstancePrincipalsAuthenticationDetailsProvider.
  5. Policy, scoped to the bucket and the verbs actually used: Allow dynamic-group invoice-sa-dg to read objects in compartment prod-apps where target.bucket.name = 'invoice-archive' plus a separate use/manage objects statement for invoice-rendered. Do not use manage all-resources.

Verification. Confirm which principal is being used by inspecting Audit logs for the GetObject request - the principal name should be the service-account principal, not an instance OCID. Then run a negative test: exec into a pod from one of the two restricted workloads and attempt the same read; it must be denied. A final positive test after rotating the pod (new token) proves the identity is not tied to the node lifecycle.

3. What risks does the migration itself introduce, and how would you sequence it?

Read the worked answer

Risks.

  • Silent fallback. If the SDK still has an instance-principal provider on the classpath, the app may succeed for the wrong reason, then fail on a different node pool - masking a broken workload identity setup.
  • Token/pod lifecycle. Credentials are tied to the pod and rotate; long-lived HTTP clients or cached signers created at startup may hold stale credentials.
  • Scope drift. The dynamic group rule is compartment-wide; without a policy that names the bucket, "service accounts in prod-apps" is broader than the original intent.
  • Cutover coupling. Changing identity and changing code in one deploy makes rollback ambiguous.

Sequencing.

  1. Provision the dynamic group, policy and test principal in a non-production namespace first, and prove the negative case (other workload denied).
  2. Remove the instance-principal provider from the code path and add an explicit startup check that logs which principal is in use.
  3. Deploy to one replica of invoice-svc (canary) behind the existing service; watch 401 rates and Audit logs for five to ten minutes, then roll out.
  4. Only after invoice-svc is stable, audit other services that were relying on the old node-level grant and either migrate them or explicitly re-grant - otherwise you have fixed one service and created a latent incident for the rest.
  5. Finally, narrow or delete the old instance-principal rule so the wide grant cannot be silently reused.