Unannounced CRM Schema Change and Silent Region Corruption

Restoring service and fixing the cause are separate. Pointing dbt at the previous partition may unblock the three failing models, but it does not restore correctness: the previous partition is 26 hours old and already contains the N/A region values, so the fourth model's 38% null-equivalent problem remains, and consumers get stale data with no visible signal. A better immediate action is to pin the last known-good partition by content, not by recency-verify row counts, schema, and region value distribution-then run a targeted compatibility layer: create a view that aliases account_tier__c to account_tier and maps N/A to NULL, and fix the positional reader by switching to named-column Parquet reads. Publish a data quality incident note stating which models are stale, which are patched, and the expected refresh time. The root cause is uncontrolled upstream schema change combined with readers that depend on column order and on implicit null semantics. Restoring the old partition without a correctness check just moves the failure downstream.

Land raw files immutably, then validate against a registered contract before any model reads them. The contract should declare column names, types, nullability, and accepted sentinel values per column; region should explicitly reject N/A, UNKNOWN, and empty strings rather than letting them pass. Run validation in a pre-load step: if the schema is compatible, promote to a curated prefix; if not, write the file to a quarantine prefix with the validation report and alert the owner. Never let a reader infer schema from position-use named columns and fail on missing required fields. Version the contract and require the vendor to announce changes; if the vendor will not, generate a contract from observed schema and treat deviations as incidents. For additive changes such as lifecycle_stage_v2, allow them only as nullable with a default and a documented deprecation window for the old name. Add a canary model that selects from the newest partition and asserts expected distributions on key columns; this catches semantic changes like the N/A substitution that pass schema validation but corrupt values.

The summary should separate trigger, mechanism, and control gap. Trigger: an unannounced vendor release. Mechanism: a renamed column broke two models, a new column shifted positional reads in a legacy loader, and a sentinel-value change turned missing regions into the literal N/A, which passed type checks and produced 38% non-null-but-invalid values. Control gap: no contract, no schema registry, schema inferred by position, and no distribution assertions, so the failure was either loud or silent but never correctly classified. Quantify impact by the share of revenue or active accounts in the affected region bucket before and after the change, not by null count alone; compare the last seven days against the prior four weeks and report the difference in the metric that uses region, such as regional ARR or churn. State the interim treatment, the backfill plan, and the owner for the contract. Avoid attributing the incident to the on-call engineer; the actionable cause is the missing validation boundary.