The Control That Passed but Didn't Work
Cobalt Health Systems runs a quarterly control testing program covering 210 key controls. In Q2, testers sampled 25 access-removal tickets and found 24 completed within the 48-hour SLA, so the control was rated 'effective.' In Q3, an incident review revealed that a terminated administrator retained system access for 19 days and exported 4,000 patient records. Investigation showed the control tested only whether tickets were closed on time, not whether access was actually removed. In 30% of sampled tickets, the account remained active despite the ticket being marked complete. The control owner says the testing methodology was 'as designed.' The audit committee asks why a control rated effective failed to prevent the incident.
1. Diagnose the design versus operating effectiveness issue here.
Read the worked answer
This is a control design deficiency, not an operating effectiveness failure. The control as designed measures ticket timeliness - a proxy - rather than the outcome it exists to achieve: access removal. A well-designed control would reconcile the termination feed against active directory accounts and confirm the account is disabled, not just that a ticket was closed.
Because the design is wrong, operating effectiveness testing was measuring the wrong thing efficiently. 24 of 25 tickets on time tells you the ticketing process works; it tells you nothing about whether access was revoked. The 30% of tickets where the account stayed active is the real signal, and the test never looked for it.
The control owner's defense - 'tested as designed' - is exactly the problem. Testing faithfully against a flawed design produces false assurance. The audit committee should hear that the rating was technically correct and substantively misleading.
2. Why did the testing program not detect the design flaw, and what does that say about the program's structure?
Read the worked answer
The program tested execution against the control description, not the control against the risk. Testers were given a control statement ('access removal tickets closed within 48 hours') and verified compliance with it. No step asked whether closing a ticket actually removes access.
Structural issues:
- No outcome validation: tests stopped at the process artifact, never confirming the end state.
- No independent reconciliation: a second data source (the termination list) was never compared to the account list.
- Sampling masked it: 25 tickets is enough to measure timeliness but too small and too narrowly scoped to reveal a systemic design gap.
- No challenge step: nobody asked 'if this control works perfectly, does the risk go away?'
A program that only checks conformance to its own descriptions will certify broken controls indefinitely. Design effectiveness needs to be assessed separately, by someone asking whether the control, if operating perfectly, would actually mitigate the risk.
3. Recommend remediation for both the control and the testing program, and identify how you would report this to the audit committee.
Read the worked answer
Control remediation: replace the ticket-timeliness control with an outcome-based reconciliation - automated comparison of the HR termination feed to identity store accounts, with exceptions escalated and a target of zero active accounts 24 hours post-termination. Retain ticket timeliness as a supporting metric, not the primary control.
Program remediation:
- Add a design effectiveness assessment before any control enters the testing population.
- Require outcome verification for controls whose purpose is a system state change.
- Introduce a challenge question in test design: 'If this control operates perfectly, is the risk actually mitigated?'
- Re-test the 210 controls for design adequacy, prioritizing access, change, and financial controls.
Audit committee reporting: state plainly that the Q2 'effective' rating provided false assurance due to a design deficiency, that 30% of sampled removals failed in practice, and that the incident was a consequence. Disclose the population at risk, the remediation plan with owners and dates, and the program-level fix. Avoid framing it as a testing error alone - the committee needs to understand the assurance gap, not just the incident.