Data Analytics for Revenue Leakage Detection
Meridian Health Distributors processes 400,000 sales invoices per year across 12,000 customers. Internal audit runs a rule-based test flagging invoices where the unit price is below the contracted price by more than 2%. The rule generates 18,000 exceptions annually, of which only 600 are true pricing errors; the rest are legitimate discounts, rebates, or contract amendments not captured in the pricing master. Investigators can review 1,500 exceptions per year. Separately, a clustering model on invoice-level features (price deviation, order size, customer segment, payment terms) identifies 2,000 invoices in a high-risk cluster; 40% of those are confirmed errors. The pricing master has 3% of records with missing effective dates, and 8% of contracts were amended mid-year without updating the master. Management wants to 'just automate the rule and email all exceptions to sales.'
1. Evaluate the precision and recall implications of the rule-based approach versus the clustering model, and explain why 'email all exceptions to sales' is a poor control design.
Read the worked answer
Rule-based precision. 600 true errors out of 18,000 exceptions = 3.3% precision. With 1,500 reviews, expected true errors found ≈ 1,500 × 3.3% = 50. The rule is noisy because it ignores legitimate discounts, rebates, and amendments-i.e., the data lacks the context needed to distinguish error from valid pricing. Recall is unknown without a full population of true errors, but if 600 is the full error population and reviewers only see 1,500 of 18,000 flags, recall is severely limited unless the 600 are concentrated in reviewed items.
Clustering precision. 2,000 invoices, 40% confirmed errors = 800 true errors. Precision is 40%, roughly 12× the rule. If investigators review only the cluster, they find 800 errors using 2,000 reviews (or fewer if prioritized). The clustering model captures multivariate patterns the single rule misses-e.g., combinations of price deviation, segment, and payment terms.
Why 'email all exceptions' fails. (1) Alert fatigue: 18,000 emails to sales will be ignored; 96.7% are false positives. (2) No accountability: sales has an incentive to explain away exceptions, not investigate them. (3) No feedback loop: without labeling outcomes, the model cannot improve. (4) Control weakness: a control must be performed by someone independent and documented; mass emailing is not a control. Better design: route the clustering model's high-risk cluster to internal audit, use the rule as a secondary screen, and require sales to attest to legitimate discounts with documentation.
2. Recommend data-quality remediation and a monitoring design that improves both precision and coverage over time.
Read the worked answer
Data-quality remediation. (1) Pricing master: enforce mandatory effective dates and a validation rule blocking invoices when the effective date is missing; backfill the 3% of records. (2) Contract amendments: integrate the contract management system with the pricing master so mid-year amendments (8% of contracts) automatically update prices; add a reconciliation control comparing amended contracts to master records monthly. (3) Discount and rebate data: capture approved discounts and rebates in a structured table so the rule can net them out, which should sharply reduce false positives.
Monitoring design. (1) Tiered triage: use the clustering model to prioritize (high-risk cluster first), then apply rules within the cluster. (2) Threshold tuning: raise the rule threshold from 2% to a level calibrated on labeled outcomes to balance precision and recall. (3) Feedback loop: log investigator dispositions (error vs. legitimate) and retrain the model quarterly; track precision, recall, and dollars recovered as KPIs. (4) Coverage: sample from the unflagged population periodically to estimate missed errors (recall), since the model only scores what it sees. (5) Governance: assign investigation to internal audit, require documentation for overrides, and report results to the audit committee. Expected outcome: reviewing 2,000–3,000 prioritized items could surface 800+ errors versus ~50 under the current rule, while data fixes reduce false positives and free investigator capacity.