340 Rules, 62% Noise, and a Rule That Is Arguably Correct
Atlas Logistics runs a warehouse-based data quality program with 340 active rules across 12 domains. Last quarter, 62% of alerts were closed as false positives or duplicates, and the on-call data engineer now triages about 150 alerts per week, mostly by silencing them. A new rule flags delivery_postcode as null for 7.3% of shipment records following a checkout form change; four downstream routing and customer-notification reports use the field. Product says postcode is optional for pickup-in-store orders, which are about 9% of volume, and that the release is behaving as designed. A stale rule on a deprecated table has fired daily for nine months. Separately, an executive dashboard shows 99.2% completeness for shipment records, but its rule excludes nulls from the denominator.
1. How would you redesign the rule portfolio given 340 rules, 150 alerts per week, and 62% noise? What would you deliberately avoid?
Read the worked answer
Quantify before cutting. For each rule, measure alerts fired, alerts actioned, time to resolve, and the downstream asset affected. A 62% false-positive rate means roughly 57 of 150 weekly alerts are actionable - the program is not under-detecting, it is over-reporting, and the silencing behavior is rational, which is the real emergency because it trains people to ignore alerts.
Redesign along four lines. Tier rules by critical data element and consumer: a rule on a field feeding customer notifications outranks a rule on a field with no readers. Retire aggressively - the nine-month stale rule is an ownership failure, not a rule problem, and it should have been deleted the day the table was deprecated. Consolidate duplicates so one defect produces one alert with one owner. Route alerts to the data owner responsible for the fix, not to a central on-call engineer who cannot change the source system.
Add expected-condition suppression with an expiry date and a named approver, so known business exceptions do not recur as noise but also cannot be forgotten forever.
What to avoid: raising thresholds to reduce volume. It lowers alert counts while lowering detection, and it hides the 7.3% postcode issue rather than resolving it. Also avoid a big-bang rewrite of all 340 rules; retire and re-tier in tranches, measuring precision weekly, targeting above 70% actionable within a quarter.
2. What is the correct response to the postcode rule - suppress it, keep it, or something else?
Read the worked answer
Neither suppress nor keep as-is. The rule as written is measuring two different populations against one threshold, which guarantees a permanent false positive of roughly 9%. That is a specification defect, not a data defect.
Resplit the rule by order type. For pickup orders, null postcode is expected and should be excluded from the rule with a documented exception, owner, and review date. For delivery orders, null postcode is a genuine defect because routing and notification depend on it, so the rule should alert above a small baseline (for example, a null rate above 0.5% of delivery orders, set from the trailing baseline). Add order_type as a dimension in the DQ result so the trend is visible, and set severity high given four downstream consumers.
Then check the consumers. If the routing and notification reports do not filter by order type, the real incident is downstream: they may be dropping pickup shipments or sending notifications with no address. That is a higher-priority finding than the null rate itself.
Finally, treat this as a change-management trigger: a checkout form release changed the data profile and no one told the DQ team. Add a lightweight notification hook from product releases that touch captured fields, plus a regression check on the affected rules. Silencing the rule would have hidden a legitimate defect in the delivery population.
3. The 99.2% completeness figure excludes nulls from the denominator. Diagnose the measurement problem and propose governance so it does not recur.
Read the worked answer
A completeness measure that excludes nulls from the denominator is circular: it reports the proportion of non-null values that are non-null. It will read near 100% on almost any dataset and it will not move when data degrades - which is exactly what an executive dashboard must never do.
Correct the definition: completeness equals the count of records with a non-null value in a required attribute divided by the count of records where that attribute is required. "Required" must be defined per attribute and per segment - postcode is required for delivery shipments and not for pickup, so the dashboard should show completeness for the delivery population, with the pickup exclusion visible as a stated scope rather than an invisible filter.
Governance: publish one definition per DQ dimension, certified once and reused; store rule logic in version control with author, change reason, and approval; any rule change that feeds a published metric triggers a downstream impact review before it deploys. Add an audit view showing rule changes over the last 90 days, because a metric that silently improves after a rule edit is a red flag.
Report both a technical score (by attribute, all rows) and a business score (by required-attribute population) so operations and executives see related but honest numbers. Then correct the 99.2% figure, state the correction openly, and treat the original rule as an incident with a root cause, not a quiet fix.