Auditing Your Own Work, Reviewing Nothing
Vantage Health (900 staff) has outsourced internal audit to the same consultancy that designed and implemented its ISMS. One audit was performed 11 months ago; it covered the IT department only and sampled 5 of the 60 applicable controls. Of 22 findings raised, none has been closed. The internal audit report is signed off by the CISO. Management review meets quarterly with a standing agenda of incident statistics and the information security budget, 20 minutes allotted, and no record of decisions or actions. The only defined metric is 'number of security incidents per month', which the CISO reports as falling steadily, after the service desk stopped logging minor events.
1. Assess the internal audit arrangement against clause 9.2, including impartiality, programme scope and follow-up.
Read the worked answer
Impartiality (9.2.2). Auditors must be objective and impartial, and must not audit their own work. A consultancy that designed and implemented the ISMS cannot objectively audit it: this is self-review in substance even if different individuals attend. The CISO signing the report compounds it, because the CISO is also the owner of most audited activity.
Programme (9.2.2). The programme must be planned with frequency, methods, responsibilities and requirements, taking into account the importance of the processes, changes affecting the organisation, and the results of previous audits. A single audit in 11 months, confined to one department, ignoring change and prior results, does not meet that.
Scope and criteria (9.2.2). Each audit needs defined objectives, scope and criteria. There is no evidence that clauses 4–10 were audited at all; 5 of 60 controls is too thin a sample to support a conclusion about control operation.
Reporting and follow-up. Findings must be reported to relevant management, and nonconformities handled under 10.2 - correction, root cause analysis, corrective action, review of effectiveness. Twenty-two open findings with no closure means the audit produced no improvement.
Options: use a different provider, or build a pool of competent auditors from other functions and cross-audit with a partner organisation; retain a technical specialist where competence is lacking. Impartiality, not independence from the company, is the requirement.
2. Which management review inputs are missing, and what should the outputs look like?
Read the worked answer
What is present: incidents and budget. That covers, at best, part of one input.
What 9.3.2 expects as inputs:
- Status of actions from previous management reviews - currently there is no record to review, which is itself the finding.
- Changes in external and internal issues relevant to the ISMS (new regulation, new customers, new sites, the outsourcing of audit).
- Feedback on ISMS performance, including trends in nonconformities and corrective actions, monitoring and measurement results, and audit results.
- Whether information security objectives have been met.
- Results of risk assessment and status of the risk treatment plan.
- Opportunities for continual improvement.
- Input from interested parties (the health regulator, patients, NHS partners).
- Adequacy of resources.
Outputs (9.3.3): decisions related to opportunities for continual improvement and any need for changes to the ISMS. These must be recorded - 7.5 requires documented information as evidence of the results of management review, so 'ISMS reviewed - OK' in a slide deck is not sufficient.
Judgement call: the answer is not to lengthen the meeting to three hours. It is to send a fixed input pack in advance, allocate decisions to named owners with dates, and start the next meeting by reviewing the previous decisions. Twenty minutes with three recorded decisions beats a two-hour briefing.
3. Critique the incident metric and propose how monitoring and measurement should be determined under clause 9.1.
Read the worked answer
Critique:
- The definition changed mid-stream. The service desk stopped logging minor events, so a falling count cannot be read as improving security. The trend is an artefact of the measurement method - the classic failure of a single volume metric.
- It measures reporting activity, not security. A drop can equally indicate under-reporting, which is a cultural failure.
- No criteria, method, timing or owner are documented for the metric, so it cannot be evaluated.
What 9.1 requires: determine what needs to be monitored and measured (including information security processes and controls), the methods for monitoring, measurement, analysis and evaluation - as applicable, to ensure valid results - when to perform it, who does it, when results are analysed and evaluated, and who they are reported to. Documented information as evidence of the results.
A workable set for Vantage Health:
- A small number of outcome measures: proportion of high-severity incidents detected internally rather than by a third party; mean time to contain; percentage of critical systems with tested recovery.
- Control effectiveness measures: proportion of privileged accounts reviewed on schedule; proportion of suppliers with current assurance.
- Process measures: internal audit findings closed within agreed timescales; percentage of staff completing role-relevant awareness.
- Stability rule: the definition of any metric may only change at a stated point, with the change recorded, so trends remain interpretable.
Reasoning: the auditor's question is not 'is the number going down?' but 'how do you know this measure tells you whether your controls work?'