Two MRR Numbers and a Silent Eight-Percent Drop

Freeze and measure before changing anything. Another silent edit on top of this one makes the audit impossible.

Hour 1–4: pin the state Tag the current dbt release, record the last known-good commit, and stop scheduled deploys to the affected models. Announce that the dashboard is under review so people stop re-sharing the number.

Hour 4–12: reproduce the 8% Run the pre-change and post-change versions of the model against the same source snapshot and diff the output. Then classify:

• If rows disappeared because multiple subscriptions per account were legitimately collapsing, the join grain is wrong - the fix removed duplicates by deleting real subscriptions, and MRR is understated.
• If the duplicate rows were genuine duplicates, the change was correct and the 8% is the value of a bug that had been inflating the metric. Evidence: count subscriptions per account before and after; inspect ten vanished rows by hand. On a monthly-and-annual book, an account routinely holds several concurrent subscriptions, so account-grain collapse is the more likely verdict - but verify, do not assume.

Hour 12–24: explain the $300k This is almost certainly not an error. MRR is recurring subscription value at list; Finance's figure is recognised revenue, net of discounts and credits, prorated, and deferred across annual plans. Neither is wrong.

Hour 24–48: publish a bridge List MRR → less discounts → less credits → deferral timing → recognised revenue. Line by line, with the residual stated. If the residual is not zero, that residual is the real data problem.

Controls follow from the failure modes observed: silent row loss, definitional divergence, and detection by executive question.

Critical data elements Define MRR, active subscription, logo churn, revenue churn and customer churn, each with an owner, a stated purpose, grain, formula and permitted values. Resolve the churn divergence explicitly - cancellations versus non-renewals differ by whether an expiry counts, and that choice must be recorded once, not made per team.

Tests, by failure mode

• Row loss: a volume test on the subscription model against a rolling baseline, plus uniqueness and relationships on subscription_id to dim_account. The missing control here was a row-count assertion, not a sophisticated one.
• Silent drops: not_null on every key and amount; accepted_values on status; a referential test from every fact to the account dimension.
• Staleness: source freshness on the subscription feed with a hard threshold.
• Definitional drift: a reconciliation test comparing MRR to Finance's revenue schedule within a stated tolerance band (for example ±0.5%, with every residual explained), run every period.
• Anomaly: alert when day-over-day MRR moves beyond a threshold that reflects normal volatility.

Service levels Freshness: available within 4 hours of source close. Completeness: at least 99.5% of source subscription records. Accuracy: within stated tolerance of the Finance reconciliation. Availability and a published known-issues page.

Detection metric Mean time to detect, measured from defect introduction to internal ticket. Today it is three weeks and an executive question; that number is the programme's headline KPI.

Test the 170 models in proportion to what they feed, not in alphabetical order.

Prioritisation

1. Trace lineage from the five metrics that appear in board or investor reporting. In practice this will touch 15–30 models, not 170.
2. For each, add the cheapest test that would have caught a known past incident: row-count, uniqueness, not_null, referential. Wire them as blocking on the deployment path.
3. Add source freshness on the three sources those models depend on.
4. Leave the remainder untested but visibly marked, with an owner assigned by domain. Untested and unowned is the dangerous combination; untested with an owner is a backlog item.

Change management that survives the departure

• Ownership encoded in the repository per model directory, so review cannot be skipped by absence.
• A required CI run of the affected models and their downstream dependents on every pull request.
• Contract tests on sources so producer changes fail in the producer's pipeline, not in the CEO's dashboard.
• A metric layer, so dashboards consume governed metrics rather than raw models; most silent breakages happen in the last join before the chart.
• Blameless post-mortem for the 8% incident, published, with the actions above as its output.

Defending it to an executive Do not argue for 170 tests. Argue that the last defect reached the CEO unanswered for three weeks, and that the proposed work reduces that to hours for the five numbers the board sees. Attach a cost: the analytical time already spent on manual reconciliation. Executive patience follows visible coverage of their own metrics, not model counts.