Shared Tokens, Orphaned Owners and Bronze PII
Arcturus Bank has three IDMC orgs - Development, Test and Production - and one Databricks account with two Unity Catalog metastores, one per region. A hundred and twenty developers share a single "svc_etl" Databricks personal access token stored in a mapping parameter, and most production connections are owned by a contractor who left eight months ago. Last month a token expired without warning and production stopped for six hours. Analysts in the EU region hold SELECT on the bronze schema because "they needed it during the migration", so they can read raw account numbers. Developers can log into the Production IDMC org with their Development credentials because both orgs map the same identity-provider group. Audit also found 22 service accounts holding Metastore Admin.
1. Redesign authentication and secret handling between IDMC and Databricks. What replaces the shared PAT, and how do you prevent another silent expiry outage?
Read the worked answer
Replace the shared PAT with per-environment service principals and OAuth. Create a Databricks service principal per environment and per function - ingestion, deployment - and authenticate with OAuth machine-to-machine using the client ID and secret, or a federated credential where available. PATs have a bounded lifetime and, in this design, no accountable owner; that is precisely why the expiry took production down for six hours.
Secret handling:
- Keep the client secret in a vault - IDMC secure connection parameters backed by CyberArk or the cloud secret manager - referenced by the connection, never typed into a mapping parameter or exported in mapping XML.
- Rotate on a schedule with an overlap window: create the new secret, update the connection, run one mapping as a test, retire the old secret.
- Alert at 30, 14 and 7 days before expiry. The outage was an observability failure as much as an authentication failure, because nothing watched expiry.
Ownership. The departed contractor left production connections orphaned. Transfer ownership to an operational group or service account, audit for ownerless connections quarterly, and attach ownership transfer to the offboarding checklist so it cannot be skipped again.
Least privilege on the Databricks side. The ingestion principal needs USE CATALOG, USE SCHEMA and MODIFY on bronze and silver - not Metastore Admin. Grant EXECUTE only where a pipeline calls a function.
2. Fix the bronze PII exposure and the Metastore Admin sprawl. Explain why your controls survive a user who bypasses the curated view.
Read the worked answer
Bronze exposure, in three parts:
- Revoke SELECT on the bronze schema from analyst groups; leave it with the ingestion service principal and data engineering. Having needed it "during the migration" is not a grant basis.
- Publish silver and gold with Unity Catalog column masks on account number, national ID and other direct identifiers, plus row filters for region. Masks and filters are applied at query time, so a notebook that bypasses the curated view still receives masked values - which is the property a masking view does not have.
- Tag columns (PII, restricted) and use those tags to drive review, and attribute-based policies where available. Tags are also what make a quarterly access review tractable instead of guesswork.
Metastore Admin sprawl. Twenty-two holders is a finding by itself. Reduce to two break-glass accounts with MFA, vaulted credentials and alerting on use. Separate account admin (billing, workspace lifecycle) from metastore admin (data privileges); most of the 22 almost certainly needed neither.
Grant hygiene. Privileges are hierarchical: USE CATALOG and USE SCHEMA must precede any meaningful SELECT, and revoking SELECT while leaving USE in place invites lateral access to other objects. Review grants on groups, reconcile group membership from the identity provider rather than workspace lists, and alert on any grant change in the production catalog.
3. How do you separate Development from Production so a developer can neither deploy nor run ad hoc queries there, while still being able to support production incidents?
Read the worked answer
Make separation structural, not procedural.
- Separate IdP group mappings per IDMC org. Development credentials working in Production means the same group is mapped to roles in both orgs. Map a development group only to Development roles and a small production-ops group to a narrow Production role; remove the overlap entirely.
- No routine human write access to Production. Developers get read-only plus log access for troubleshooting. Deployment runs through a dedicated deployment service principal holding create and modify privileges and nothing else.
- Promote artefacts, not credentials. Export mappings, taskflows and connections through a versioned repository, then import with a script that rewrites connection references to the production service principal. Validate in Test on production-shaped data with the production rule set.
- Mirror it in Databricks. Separate catalogs or schemas per environment, with CREATE granted to the developer group only in development. Production jobs execute as the deployment principal, and interactive cluster use in production is limited to the ops group.
- Detective controls. Alert on any Production IDMC login outside the ops group, any grant change in the production catalog, and any table modification that did not come from a registered job.
Separation you cannot observe is separation you cannot prove. The alerting is what turns the policy into an auditable control rather than a document.