Macro-induced CI explosion and unversioned production deploys

state:modified compares the compiled representation of each node, not its source file. A macro change alters the compiled SQL or config of every model that calls it, so all of them legitimately register as modified - the selector is behaving correctly. Selection is then amplified by indirect selection: tests attached to modified models pull in their parents, and state:modified+ adds every downstream node. Before assuming, run dbt ls --select state:modified+ with the production manifest and read the list; the 620 is usually a mix of true macro dependents and indirectly pulled parents.

Redesign in four parts.

1. Bound the blast radius at the source. Move shared macros into an internal package with semantic versioning. A macro change becomes a version bump with an explicit impact list reviewed once, rather than an invisible edit inside the repo.

2. Split selection. Run dbt build --select state:modified+1 for the PR gate, and schedule state:modified+ as a post-merge job against staging where a 55-minute run is acceptable. Use --indirect-selection=cautious so tests do not drag in unrelated parents.

3. Defer to production. --defer --state prod_manifest lets unmodified upstream models resolve to production tables, so the CI build only materializes what changed.

4. Parallelize and cache. Two or three CI jobs split by selector, size-M warehouses, and no full-refresh in PR CI. Fifteen minutes is achievable for anything under ~80 models.

Keep an escape hatch: a ci:full label that runs everything for changes to dim_account, dim_customer, and the macro package - those are the nodes where a shallow check is not enough.

The failure was not the removed column; it was that nothing between merge and production could refuse the change or undo it.

Build once, promote the artifact. On merge to main, a staging job builds into a staging schema and writes manifest.json to versioned object storage keyed by commit SHA. Production runs dbt build --select ... --defer --state <staging manifest> so unbuilt nodes resolve to the staging or production relations you intend - the exact manifest that passed is the one that ships. Record the SHA in a dbt_invocations table and in Airflow's run metadata, so "which version is live" is queryable.

Gate the schema. Add dbt contracts to dim_ and fct_ models. A removed or retyped column then fails at build time in staging, not in a dashboard six hours later. Where contracts are too heavy, add a CI check comparing the new manifest's columns against the last production manifest and fail on removals unless a deprecation_date is set.

Rollback. dbt has no built-in rollback, so plan at the relation level: (a) build marts into versioned schemas and switch consumption with ALTER SCHEMA ... SWAP WITH or a view pointer; (b) for tables, Snowflake time travel (AT(OFFSET => -3600)) restores a pre-deploy state within the retention window; (c) dbt retry resumes a partially failed run from the last failure point instead of rebuilding from scratch.

Use model versions for the column case. dim_account v2 with latest_version and deprecation_date lets 12 dashboards migrate on their own schedule while both columns exist. Removal becomes a scheduled event, not an incident.

The gap is sensor granularity. A sensor on the previous job waits for the last model in that job to finish, so every model in core that only depended on the first ten models of raw sits idle. Add three jobs of fixed inter-job latency plus polling intervals (poke_interval of a few minutes each), and you can easily lose two to three hours to waiting on work that was already done.

Primary change: one dbt invocation. Let dbt's own DAG schedule within a layer instead of Airflow scheduling between layers. A single dbt build --select +fct_orders+ builds only what that mart needs, in dependency order, at eight or sixteen threads, with no cross-job barrier.

Where separate jobs still earn their keep. Keep boundaries where the failure semantics differ, not where the layers differ:

• A raw snapshot/ingest job, because it depends on an external extract and has different retry and alerting needs.
• A small "critical marts" job that runs first with a tight SLA, so the 12 dashboards that matter refresh early even if the long tail slips.
• A nightly job for tests, snapshots, and --full-refresh candidates, where a long runtime is acceptable.

Instrument the wait. Pull Airflow task durations and compare queue time against dbt's reported per-model times. If queue time stays above about 10% of total after consolidation, the bottleneck is warehouse sizing or thread count, not orchestration. Rewrite the sensors as dataset-driven or deferrable operators so any remaining cross-job dependency polls without holding a worker slot.