Compartment, Policy, and Identity Design for a Shared Data Science Platform
Acme Retail runs a single OCI tenancy shared by three lines of business (Grocery, Apparel, Digital) and one central data platform team. Fourteen data scientists create notebook sessions and jobs. All training data lands in one Object Storage bucket, ds-data-lake, in the platform compartment; each LOB also has its own curated bucket. Compliance requires that auditors can attribute every read of ds-data-lake to a named human or to an approved workload, that LOB compute spend be separable for chargeback, and that a Grocery scientist cannot read Apparel's curated bucket. Today scientists paste their own API keys into notebooks and download the whole lake locally; one departed scientist's key is still active. Security refuses to issue long-lived keys and wants least privilege, but the platform team does not want to approve a new policy for every new notebook. Design the target architecture.
1. Propose a compartment and policy layout that satisfies data isolation, chargeback, and auditability. Explain why a single 'allow read on ds-data-lake' policy is or is not sufficient.
Read the worked answer
Layout: a DataScience parent compartment with a child per LOB (Grocery, Apparel, Digital), and inside each child one compartment per project containing that project's notebook sessions, jobs, model deployments, and curated bucket. The shared lake stays in a separate DataPlatform compartment. Apply a defined tag (for example CostCenter=Grocery) at the LOB compartment so compute can be tagged by default and rolled up for chargeback.
Policies: grant read objects on the lake bucket to a single dynamic group that represents all approved data science workloads, so platform engineering approves one policy, not one per notebook. Grant curated-bucket access per LOB compartment and per LOB group only.
Why one broad policy is not enough for isolation: OCI IAM policies are additive and have no deny statements. You cannot write 'read the lake except the Apparel bucket'. Isolation therefore has to come from placement - Apparel's curated objects must live where a Grocery principal has no policy path to them - not from a conditional read grant. Trying to encode exclusion in a where clause leaves a hole the first time someone copies data into the lake compartment.
Audit: control-plane calls appear in OCI Audit, but object-level GETs need Object Storage access logging routed to a log group the auditors can read. Grant auditors read on logs, not on data.
2. Write the dynamic group matching rules you would use for notebook sessions and for jobs, and identify one failure mode of scoping the rule too tightly.
Read the worked answer
Use workload identity rather than user credentials:
- Notebooks:
ALL {resource.type='datasciencenotebooksession', resource.compartment.id='ocid1.compartment.oc1..grocery'}(one entry per LOB or project compartment, combined withANY). - Jobs:
ALL {resource.type='datasciencejobrun', resource.compartment.id='...'}. - Pipelines (if used):
resource.type='datasciencepipelinerun'.
Inside session code, obtain credentials with the resource principals signer rather than an API key file, so no secret is stored in ~/.oci and rotation is a non-event.
Failure mode of tight scoping: matching on resource.id for a specific session or on a compartment that a project later moves out of causes silent, total breakage - the workload simply loses the policy and returns NotAuthorizedOrNotFound at runtime, often in a scheduled run nobody is watching. Scope by compartment and resource type, keep the rules in version control, and add a smoke test that each workload identity can list the buckets it depends on.
Trade-off to state explicitly: with resource principals, any notebook in the same compartment inherits the same access. That is acceptable only because compartments are per LOB and per project; if they were not, the convenience of one policy would defeat the isolation requirement.
3. Sequence the migration away from pasted API keys, including what to do about the departed scientist's key and about attribution.
Read the worked answer
- Build the target first. Create compartments, buckets, dynamic groups, and policies; add tags for chargeback. Do not remove keys before the replacement works.
- Prove workload identity in one project. Patch a small notebook and a small job to use the resource principals signer and read a test object. Confirm the policy path is the one you think it is, not a broader leftover policy.
- Migrate in waves per LOB, moving curated data into its new compartment at the same time, because isolation depends on placement.
- Revoke. Delete the departed scientist's key now - it is a standing risk independent of the migration - and rotate any keys that remain. Then add a policy that forbids user-level auth tokens in project compartments, or at minimum detect them with a scheduled scan of
~/.ociand notebook contents. - Close the attribution gap. Resource-principal access is attributed to the resource, not the person. Keep the mapping from session OCID and job run OCID to creator, retain it beyond the resource's life (sessions and runs are deleted), and log it alongside object-access logs so an auditor can answer 'who read this' with evidence rather than inference.
Judgement point: the tempting shortcut is to keep one shared service account key in a notebook home directory. It satisfies least effort and fails least privilege, rotation, and attribution simultaneously - reject it explicitly in the design review.