dbt Analytics Engineering: Cross-team refs, PII lineage, and a docs gate that has to fit in CI

Cross-team refs, PII lineage, and a docs gate that has to fit in CI

Northwind Logistics consolidated three teams into one 780-model dbt repo six months ago. Finance, Ops, and Growth each own part of the DAG; there is no groups: configuration, so any model can ref any other, and analysts regularly build dashboards on int_ models that a team later renames or drops. Fourteen models expose customer PII (email, phone, national ID), and an internal audit has asked for proof of which published dashboards - tracked loosely in a spreadsheet - depend on those columns. Descriptions exist for about 40% of models, mostly a single line, and dbt docs is deployed only when someone remembers. Column-level metadata is inconsistent: three teams tag PII differently (pii: true, sensitive: y, gdpr: yes). The repo has 1,100 tests and a 45-minute full CI run, so any docs or metadata gate has to fit inside that budget.

1. How would you stop analysts from building on another team's intermediate models, using dbt's own project configuration rather than a wiki page? Describe the policy and the rollout.

Read the worked answer

The problem is that ref is unconstrained: the DAG records dependency but not ownership. dbt's groups and access configuration is the mechanism. Declare groups in dbt_project.yml or a properties file with name and owner, attach each model with group, and set access per layer:

access: private on every int_ model. Only models in the same group can ref it; a cross-group ref fails at parse time, in CI, before merge. That single setting eliminates the whole class of 'an analyst built a dashboard on an internal model' breakage.
access: protected (the default) on staging and most fct_/dim_ models - referenceable anywhere inside the project.
access: public only on the curated consumption layer that other projects and dashboards may depend on. Public is what makes a model an API contract: additions are cheap, removals are breaking.

Add meta for discoverability: meta: {owner: finance_analytics, tier: gold} on public models, plus a docs block stating what the model guarantees.

Rollout matters more than the YAML. Start with one team's private models and run CI: the parse errors enumerate every existing cross-team dependency, which is your migration backlog. Give a two-week window with a scheduled report over manifest.json naming offenders, then flip to private. Set group: on sources too, so dbt ls --select group:finance and --select group:finance,+ become usable for ownership-scoped runs and incident triage.

2. An internal audit wants to know which published dashboards depend on PII columns. What artifacts would you produce, and where does dbt's lineage stop being precise enough to be trustworthy?

Read the worked answer

Three artifacts, plus one honest caveat.

Normalise the metadata. Choose one convention - column-level meta: {pii: true, pii_type: email} - and record it in the source YAML, not only in marts. A CI check that greps for the three legacy spellings (pii, sensitive, gdpr) and fails on anything outside the convention prevents drift. If the warehouse is Snowflake, also attach policy_tags in the column config so masking is enforced by the platform, not merely documented.
Declare the dashboards as exposures. Each one becomes an exposures: entry with type: dashboard, owner, url, and depends_on: [ref('fct_claims')]. That converts 'tracked loosely in a spreadsheet' into nodes dbt can validate and report on.
Compute the answer from the manifest. After dbt docs generate, parse target/manifest.json; take every node with a PII meta flag, build a reverse adjacency from depends_on.nodes, and walk downstream from those nodes to the exposures that reach them. Emit one table: PII column, path, exposure, owner.

The caveat: column-level lineage is reliably available in dbt Cloud Explorer where the warehouse and plan support it. In a locally hosted docs site the lineage is model-level, so the manifest walk over-includes - every column of a model that holds one PII column is treated as PII. State that in the audit response instead of claiming column precision you do not have, and store the generated report as a versioned CI artifact each release so the answer carries a date and a commit.

3. You want a documentation and metadata gate in CI, but full CI already takes 45 minutes. What checks do you add, and what do you deliberately leave out?

Read the worked answer

A documentation gate must be fast, deterministic, and scoped so it does not block the first merge on six months of legacy debt.

Checks worth adding, all inexpensive:

Coverage on changed models only. dbt ls --select state:modified --state ./prod-manifest --resource-type model produces the changed set; a short script then reads manifest.json and fails if any of those models lacks a description or meta.owner. Scoping to state:modified turns the undocumented 60% into a backlog instead of a wall.
Exposure integrity. dbt warns at parse when an exposure points at a missing model, but warnings get lost in job logs. Fail the job on any parse warning mentioning an exposure - orphaned exposures are exactly how a governance story rots.
PII convention enforcement. Fail if a model in the public access layer has a column matching email, phone, or national-ID patterns without the pii meta key.

Deliberately excluded: prose style linting and description length rules. They generate churn without improving trust, and they will be the first thing someone disables. Also keep dbt docs generate out of PR CI - it belongs in the deploy job, publishing manifest.json, catalog.json, and the site as versioned artifacts, retaining the last several manifests so you can diff lineage between releases.

Last, put docs next to code: require that a model change and its description change land in the same PR, so a reviewer judges both together. A separate documentation sprint never happens.