Safety, Guardrails, and Responsible AI
What to know
Safety is layered: model-level behavior (training and system-prompt policies), input filtering, output validation, and human oversight. Candidates should understand prompt injection - untrusted text in documents, tool results, or user input attempting to override instructions - and mitigations: separating data from instructions, delimiting untrusted content, least-privilege tools, and validating model actions before execution.
Know that guardrails should be enforceable in code, not just requested in prose: schema validation, allow-lists, rate limits, PII redaction, content classifiers, and human-in-the-loop approval for high-impact actions. Understand the principle of least privilege for tools (read-only where possible) and that the model's confidence is not a safety control.
Traps
- Believing a system prompt alone prevents injection; untrusted content can still influence output.
- Granting broad tool permissions "just in case."
- Trusting model-generated JSON without schema validation.
- Skipping human review for irreversible or high-stakes actions (payments, deletions, external messages).
- Logging raw prompts containing secrets or PII.
- Assuming refusals are always correct or always wrong; evaluate both over- and under-refusal.
Rule of thumb
Defense in depth: constrain the model, validate every input and output in code, minimize tool privilege, and require human approval before irreversible actions.