The Repeat Finding and the Ninety-Nine Percent Metric

Answer: the previous action changed a form, not a cause, and no effectiveness check was ever defined.

Analysis:

• The action addressed a symptom. A checklist item in a release template records that someone asserted evidence exists. It does nothing about storage location, naming, retention period, who reviews the evidence, or what happens when it is missing. A 94% tick rate is a measure of form-filling, not of evidence retention - and the 6% unticked is the only interesting number.
• Root cause needs to be established, not assumed. Likely candidates to test: no defined evidence repository, so evidence lives in email and laptops; no retention requirement in the ONF, so nobody knows how long or in what form; verification performed by the same person who implemented the control, so the need for retained evidence was never felt; and release deadlines that make the checklist item the first thing dropped. Each cause implies a different fix.
• Replacement action should be structural. Define evidence types per ASC in the ASC library, provide a system of record with immutable retention, make the verifier's sign-off dependent on evidence being attached, and sample-audit a fixed number of applications per quarter to confirm retention.
• Define effectiveness before implementing. For example: a sample of 20 ASC verifications across 10 applications in the next quarter shows the required evidence retrievable within one working day, with zero unexplained gaps. Set the check date and the owner now, and have someone independent perform it.
• Treat a repeat finding as a governance signal. The same finding twice means the corrective action process is not testing itself, which is a bigger issue than the evidence gap.

Answer: a metric that counts activity is being gamed because it does not describe coverage or depth.

Redesign principles:

1. Define the unit precisely. '100% of applications' should mean a named application inventory with an owner, and 'tested' should mean the verification activities required by that application's target verification level were performed and evidenced. A single scan of a login page never satisfies any level.
2. Report coverage and depth separately. Coverage: proportion of the inventory with a current verification record. Depth: proportion of required ASC verifications that produced acceptable evidence, per application, broken down by verification method. A single blended percentage conceals exactly the failure the auditor found.
3. Add a freshness and change dimension. An application tested eighteen months ago and changed substantially since is not tested. Tie the metric to significant-change triggers, so the number reflects the current state.
4. Add a quality counterweight. Track findings reopened because evidence was rejected, and the proportion of verifications done by someone independent of the implementer. Without a counterweight, any coverage metric can be met by lowering the bar.
5. Have the metric definition owned outside the team being measured. The security team choosing its own success criteria produces the gaming; the metric definitions should be agreed with internal audit and reviewed annually.
6. Expect the number to fall when the definition tightens. Brief the executive beforehand so the drop is understood as a measurement correction, not a deterioration.

Answer: two conflicts, both fixable through reporting lines and role separation rather than headcount.

Issues:

• Internal audit reporting to the programme director compromises the function that must be able to challenge the programme. Independence is about reporting line and freedom to escalate, not about budget. Internal audit should report to the audit committee or equivalent, with an unrestricted right of access and escalation. If the organisation cannot restructure in this cycle, document the impairment and have the audit committee directly commission the application security audit.
• The head of delivery approving ASC exceptions places the person accountable for schedule and cost in the position of deciding whether controls can be waived. Exceptions should be approved by the application owner as the risk owner, with security providing assessment and the exception register reviewed by a body that does not have delivery targets. Delivery can attend; it should not decide.
• Segregation of implementer and verifier is the cheapest independence gain. With flat headcount, use peer verification between teams in different business units, and reserve a small central verifier capacity for high-criticality applications, rather than accepting self-verification everywhere.

Sequencing: exception governance and internal audit reporting first, because both are policy changes; peer verification next; and an external assessment retained for the highest-criticality applications where internal independence is weakest. Record the residual impairment in the management review so the audit sees a managed constraint rather than an unnoticed one.