Red Team and AI/ML Security
What this domain tests
This domain pairs adversary-emulation discipline with machine-learning attack surface. Expect questions on scoping and rules of engagement, technique mapping, operational security, and on how LLM-backed features fail when untrusted text meets over-privileged tools.
Engagement discipline
- Rules of engagement fix in-scope hosts, domains and cloud accounts, excluded data, permitted techniques, escalation contacts and log handling. Testing outside them invalidates the engagement regardless of findings.
- Map actions to technique identifiers so the defensive team can build detections rather than read a narrative.
- Cloud control planes record API calls independently of endpoint tooling, so host-level stealth does not hide identity activity.
ML attack surface
- Prompt injection is direct when a user supplies it, and indirect when it arrives through retrieved documents, web pages, tool output or code comments. Indirect injection is the higher-risk case because it bypasses the user's intent entirely.
- A jailbreak bypasses model policy; an injection hijacks task flow. They are different findings with different fixes.
- Excessive agency covers tool scopes, missing confirmation on destructive actions and credentials reachable by the agent.
- Insecure output handling is where model text reaches a classic sink: SQL, shell, HTML or an outbound HTTP client.
- Supply chain: pickle-based artifacts execute code on load, so prefer safetensors and verify hashes.
Worked example
An internal retrieval assistant exposes search_docs and send_email(to, body). A shared PDF contains hidden text instructing the model to email the user's recent messages to an external address when summarised. The retrieved text enters the prompt with no trust label, the model treats it as an instruction, and send_email accepts any recipient with no confirmation, so data leaves the boundary. The chain is injection to excessive agency to exfiltration. Controls sit at each hop: label retrieved content as untrusted, allowlist recipients, require human confirmation for outbound actions, filter egress, and log tool invocations as detections.
Traps that cost marks
- Labelling every odd model output a jailbreak instead of naming the mechanism.
- Treating a system prompt as a security boundary.
- Assuming deterministic behaviour; report success rates across repeated samples, not one run.
- Running active tests against production with live user data outside written approval.