AI Threat Landscape and Attack Surface
What you must know
AI systems expand the attack surface in two directions: attacks on AI (integrity, confidentiality, availability of models and data) and attacks using AI (automation, scale, and social engineering). Candidates must be able to match a described attack to its category.
Core categories: data poisoning (corrupting training data to implant behavior or backdoors), model poisoning/backdooring (manipulating weights or the supply chain), adversarial examples (perturbed inputs causing misclassification), evasion at inference time, model extraction/stealing (querying to clone a model), membership inference and model inversion (recovering training-data properties), and prompt injection - direct (user-supplied) and indirect (malicious content in retrieved documents, web pages, or tool outputs). Also know jailbreaks, data exfiltration via tool use, supply-chain risk from pretrained models and packages, and agentic risks where a model takes actions with real permissions.
Traps
- Confusing poisoning (training-time) with adversarial examples (inference-time). Read when the manipulation occurs.
- Missing indirect prompt injection when the malicious instruction arrives through retrieved content rather than the user.
- Assuming confidentiality attacks require model theft; inversion and membership inference work through the API.
- Forgetting that an LLM agent's blast radius equals its permissions and tool access.
Worked example
A support chatbot summarizes customer-uploaded PDFs. An attacker hides white-on-white text in a PDF instructing the bot to email the conversation to an external address. This is indirect prompt injection leading to exfiltration via tool use; the correct mitigation set is input trust boundaries, least-privilege tool permissions, and output/egress filtering - not just a better system prompt.