Alert fatigue, a silent column rename, and a build blocked by freshness

Alert volume is a routing problem before it is a test problem. Split it into three decisions.

1. Source tiers. Tier 1 (claims, eligibility - feeds revenue and the executive dashboard): warn_after 6h, error_after 12h. Tier 2 (operational reference data): warn 24h, error 48h. Tier 3 (marketing and other rarely refreshed extracts): no freshness configuration at all; a stale marketing CSV is not an incident.
2. Test severity. Reserve severity: error for tests whose failure makes a downstream number wrong: uniqueness on primary keys, not_null on keys and amounts, relationships to dimensions. Everything exploratory starts at severity: warn with store_failures: true into a dedicated audit schema, so the team can query the offending rows instead of paging a human.
3. Volume thresholds. Noisy tests get warn_if: '>0' and error_if: '>500', turning a binary alarm into a signal: a 60-row gap is recorded, a 60,000-row gap fails the job.

Decouple freshness from model builds. Run dbt source freshness as its own job or job step so a stale marketing feed cannot abort a dbt build partway and strand 300 models. Then a freshness failure alerts the source owner, while a build failure means transformation logic broke - two different audiences, two different pages.

Finally, prune. A test nobody has acted on in a quarter is documentation, not a control. Demote it to warn, review warn results weekly from run_results.json, and delete the ones that never fire.

Every existing test was behavioural, and this failure was structural. not_null tests ran on specific column names; when the feed renamed payer_id to payer_identifier and re-added a nullable payer_id, the source YAML still declared tests against a column that, from the model's point of view, kept existing - it just became empty. Nothing in the suite asked whether the column still carried meaning, only whether it was present.

The cheapest checks that would have caught it:

• not_null plus accepted_values or relationships on every key and foreign key column in the source YAML, not only in marts. A not_null on payer_id at the source turns 100% nulls into a failure on day one. Review that column list whenever the source team ships a release.
• A schema-drift singular test querying information_schema.columns for each critical source table and failing when the column set changes. This is the only check that catches a rename before a human notices the dashboard.
• A distribution guard on low-cardinality columns: not_null combined with a row-count threshold would have fired immediately for a column that went from 100% populated to 0%.
• dbt source freshness with a load-time loaded_at_field, which catches a feed that stops rather than one that changes shape - necessary but not sufficient here.

Nine days is a detection-latency problem. The fix is a check on shape plus a check on occupancy, not more value-level tests on unrelated columns.

Three permanently failing tests are a tax: they train everyone to ignore the channel and they hide the next real failure. Handle the gap explicitly instead of muting it.

Option A - quarantine if the cause is known and defensible. Filter the bad rows out in the staging model, materialize them in stg_claims_rejected, and test the exception instead: a singular test that fails when the excluded count exceeds, say, 100 rows in a day. The main model is now clean, its not_null can be promoted to severity: error, and a worsening upstream problem still pages someone because the count test fires.

Option B - severity plus stored results if the cause is still under investigation. Keep the rows, set severity: warn, and add store_failures: true with a view materialization in a dedicated audit schema. Run the nightly job with --store-failures so the failing rows are queryable and the data owner can self-serve. Pair it with warn_if: '>0' and error_if: '>1000' so the test escalates when the gap grows from 60 rows to 6,000.

Choose A when the exclusion is business-justified and the real fix belongs upstream; choose B while the root cause is open. What to avoid: enabled: false, which deletes both the signal and the memory of why it went away, and blanket severity: warn on key columns, which is how the payer_id failure went unnoticed. Either path should carry a meta note with an owner and a ticket link, revisited at the next quarterly test audit.