Monitoring, Auditing, and Program Effectiveness Evaluation
What is tested
Know the line between monitoring and auditing. Monitoring is ongoing and performed by management or compliance (first and second lines): transaction testing, reviews, ride-alongs, spot checks, control self-assessments. Auditing is independent, periodic, and objective, typically internal audit (third line), following a risk-based plan, documented sampling methodology, workpapers, and findings rated by severity.
Concepts to name
Program effectiveness evaluation: KPIs and outcome metrics, culture surveys, hotline and investigation analytics, training effectiveness data, benchmarking, and external assessment. The DOJ's three practical questions frame most items: is the program well designed, is it applied earnestly and in good faith, and does it actually work in practice? Also know the independence requirements for the audit function.
Traps
Activity metrics (how many trainings, how many hotline calls) measure effort, not effect; questions often reward the outcome metric. Internal audit independence is compromised if management controls the audit plan or restricts committee access. Sampling without a documented, risk-based methodology is a finding. Conflating a compliance review with an audit is a frequent distractor.
Rule of thumb
Pair every activity metric with an outcome metric: training completion rate alongside test scores and observed behavior; hotline volume alongside substantiation and repeat-issue rates.
Worked example
A CCO reports 100% training completion and calls the program effective. Better answer: add knowledge-check pass rates, a scenario-based assessment for high-risk roles, and a comparison of substantiated cases before and after the training cycle.