AI Security, Privacy, and Governance
Threat landscape
Know classic and AI-specific attacks: data poisoning (corrupting training data), adversarial examples (perturbed inputs causing misclassification), model extraction (stealing behavior via queries), evasion, and membership inference (inferring whether a record was in training data). Generative systems add prompt injection (malicious instructions hidden in content), jailbreaking, and insecure output handling where model output is executed downstream.
Privacy protections
Understand anonymization versus pseudonymization, aggregation, differential privacy (adding calibrated noise), and federated learning (training locally, sharing updates). Recognize that models can memorize training data, so sensitive data in prompts or training corpora creates exposure. Data minimization and retention limits are core controls.
Governance and regulation
Governance covers policies, roles, risk classification, documentation, audit trails, and incident response. Be familiar with the idea of risk-tiered regulation (higher obligations for higher-risk uses), transparency duties toward users, and requirements for human oversight in consequential decisions. Know that governance spans the full lifecycle, from data sourcing to decommissioning.
Worked example
A retrieval chatbot ingests user-uploaded documents. An attacker embeds instructions in a document to exfiltrate other users' data. Mitigations: treat retrieved content as untrusted, isolate instructions from data, restrict tool permissions, and log and monitor outputs.
Rule of thumb
Treat all model inputs and outputs as untrusted data, and assume anything in training or prompts may eventually surface.