AI Security: Technical Security and Governance
This domain tests how you secure AI systems across the lifecycle and how technical controls connect to governance. Expect questions on threat taxonomies, adversarial machine learning, LLM-specific risks, supply-chain security, assurance, and the management-system controls that make security repeatable.
Core rules and concepts
- Use a current taxonomy. The OWASP Top 10 for LLM Applications 2025 covers LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, and LLM10 Unbounded Consumption.
- Distinguish attack families. Evasion perturbs inference inputs; poisoning manipulates training data; backdoors use trigger patterns; model extraction steals functionality through queries; membership inference tests whether a record was in training data; model inversion reconstructs features. Prompt injection can be direct or indirect; jailbreaks bypass safety behaviour.
- Map threats to MITRE ATLAS and govern risk with ISO/IEC 42001 (certifiable AI management system), ISO/IEC 23894 (AI risk guidance), and the NIST AI RMF functions Govern, Map, Measure, Manage.
- Apply defence in depth: model inventory, data provenance, third-party assessment, ML-BOM, secure SDLC, red teaming, input and output filtering, least privilege for tools, human approval for high-impact actions, rate limiting, sandboxing, egress control, logging, and incident response.
Worked example
A bank's support assistant uses retrieval-augmented generation over internal documents and can send email. A malicious document tells the model to email customer data. This is indirect prompt injection (LLM01), enabled by a vector and embedding weakness (LLM08) and excessive agency (LLM06). Reasoning: retrieved text is data, not an instruction; the email tool must enforce user-scoped authorisation and require human approval; ingestion must validate provenance; output DLP blocks sensitive data. Fine-tuning alone does not fix the trust-boundary failure.
Traps that cost marks
- Calling poisoning evasion; evasion happens at inference.
- Treating prompt injection as a model bug fixed by alignment; it is an architectural trust-boundary failure.
- Assuming ISO/IEC 42001 certifies model accuracy or security; it certifies the management system.
- Forgetting supply chain: base model, adapters, plugins, vector store, data pipelines.
- Assuming model output is safe because it is internal or because the model is aligned.