The Monitoring Sample That Flattered Itself
Your compliance monitoring team tested 25 of 4,000 customer onboarding files last quarter and reported a 96% pass rate on identity verification. The sample was drawn from files flagged as "complete" by the operations system. Two weeks later, an internal audit found that 300 files in a separate queue labeled "pending documentation" had never been verified, and 40 of them belonged to high-risk customers. The monitoring lead says the 96% was accurate "for the population tested" and that the pending queue was operations' responsibility. The chief risk officer wants to know whether the monitoring program failed.
1. Evaluate the monitoring team's defense and identify the design flaws.
Read the worked answer
The defense is technically true and substantively wrong. Testing only files the system marked complete guarantees a favorable result because the exclusion criterion is the very condition that creates risk. The population definition, not the sample size, is the flaw: the pending queue was part of onboarding and should have been in scope. Drawing from a pre-cleared population is a form of selection bias that makes the pass rate uninformative about the control's actual performance.
Additional flaws: the sample of 25 from 4,000 is thin for detecting low-frequency failures in high-risk segments, and the team did not stratify by customer risk. Saying the queue was operations' responsibility confuses ownership of remediation with ownership of assurance; monitoring exists precisely to catch what operations misses.
2. Recommend how to restructure the monitoring program and what to report to the CRO.
Read the worked answer
Restructure: (1) define populations from the full onboarding universe, including pending and exception queues; (2) stratify samples by risk tier so high-risk customers are oversampled; (3) document the population source and reconcile counts to the system of record before sampling; (4) add a coverage metric showing what percentage of the universe was eligible for testing; (5) set escalation triggers when a queue grows beyond a threshold.
Report to the CRO: acknowledge the program failure plainly, quantify the exposure (300 unverified files, 40 high-risk), state that the 96% figure should be withdrawn or heavily qualified, and present a remediation plan with owners and dates. Recommend re-testing the full pending queue immediately and a lookback on prior quarters to see whether the same blind spot existed earlier.