The Metric That Looked Great
An internal audit department reports a 'QAIP health score' to the audit committee each quarter. The score is a weighted average of four metrics: engagement review completion (40%), average review turnaround time (30%), number of findings closed (20%), and stakeholder satisfaction survey results (10%). For four consecutive quarters the score has been 92–95 out of 100, and the CAE cites it as evidence of a mature QAIP. However, an internal assessment found that reviewers were closing review notes with minimal evidence, turnaround time was gamed by reviewing only low-risk engagements first, and the satisfaction survey had a 12% response rate concentrated among auditees who received favorable reports. The CAE is reluctant to change the score because 'the board likes it.'
1. Diagnose why the health score is misleading. Address both the metric design and the behaviors it incentivizes.
Read the worked answer
The score is a composite of activity and perception measures, with no outcome or quality measures, and it is being gamed.
Design flaws:
- All four inputs measure activity (reviews done, time taken, findings closed) or satisfaction, not quality of assurance.
- Weighting review completion at 40% rewards volume over depth - a one-line 'looks fine' review counts the same as a substantive one.
- Turnaround time rewards speed, which incentivizes reviewing easy engagements first and rushing hard ones.
- 'Findings closed' can be inflated by closing findings prematurely or by avoiding difficult findings.
- Satisfaction at 10% with a 12% self-selected response rate is noise, not signal.
Behavioral effect (Goodhart's law): when a measure becomes a target, it ceases to be a good measure. The team optimized the score, not the quality. A high, stable score across four quarters is itself a warning sign - real quality programs show variation and occasional adverse findings.
2. Propose a redesigned set of QAIP indicators that would be more defensible to the audit committee.
Read the worked answer
Shift the balance toward quality, outcome, and independent evidence:
Quality of review (not just completion):
- % of reviews with documented substantive conclusions (sampled and validated).
- Independent re-performance or second-level review of a sample of engagements.
Coverage and risk alignment:
- % of high-risk engagements reviewed (not just count).
- Review coverage by engagement type, including co-sourced work.
Outcomes:
- Findings validated as closed through follow-up testing (not self-reported closure).
- Repeat findings rate; root-cause analysis completion.
Stakeholder input (fixed):
- Survey with a target response rate and stratified sampling; report response rate alongside results.
Program health:
- Remediation timeliness for QAIP deficiencies; internal assessment completion.
Present these with baselines and trends, and explicitly tell the committee that a lower, more variable score with honest adverse findings is a better signal than a flat 95.
3. How should you advise the CAE to communicate the change to the board without appearing to admit prior misreporting?
Read the worked answer
Frame it as maturing the program, not as a confession - but be transparent about the limitations found.
- Explain that the prior score measured activity and perception, and that the internal assessment identified gaps in how those measures could be interpreted.
- Present the new indicators, the rationale, and the baselines; note that scores may initially look lower because they now measure quality.
- Disclose the specific issues found (review depth, gaming of turnaround, survey bias) and the remediation actions.
- Avoid blaming the board for 'liking' the old score; the CAE owns the metric design.
The credibility risk of continuing to report a misleading score is far greater than the discomfort of explaining a better one. The board's confidence depends on the CAE's willingness to report bad news accurately.