Safety, Security, and Responsible AI Governance
Threat model first
Understand the attack surface: prompt injection (direct and indirect via retrieved content or tool output), jailbreaks, data exfiltration through tool calls, insecure output handling (model output rendered as HTML or executed), and training-data or context leakage. Indirect injection through documents and web content is the classic trap - retrieved text is untrusted input, not instructions.
Defenses in depth
Know layered controls: input and output filtering, instruction/data separation with delimiters, least-privilege tool scopes, allow-lists for domains and actions, sandboxing for code execution, human approval for high-impact actions, and rate limits. Never rely on a single system prompt as a security boundary. Treat model output as untrusted when it reaches downstream systems.
Governance and responsible AI
Understand policy and process: data classification and handling, PII redaction, retention and deletion, consent, audit logging, model and prompt change management, and documented risk assessments. Know fairness and transparency concerns: bias in outputs, disclosure that users are interacting with AI, and human oversight for consequential decisions. Map controls to frameworks and regulations at a conceptual level rather than memorizing clause numbers.
Traps
Options like "the system prompt forbids it, so it's safe" or "filter the input and you're done" are wrong. Watch for conflating authentication (who you are) with authorization (what the agent may do). Another trap: logging full prompts and responses containing PII without redaction.
Rule of thumb
Assume the model can be manipulated: enforce security in code around the model (least privilege, validation, approval gates), and treat every retrieved or tool-returned string as untrusted.