Trustworthy AI
What to know
Trustworthy AI for LLMs spans alignment, safety, privacy, fairness, and transparency.
Alignment covers instruction tuning, RLHF/DPO, and the helpfulness-harmlessness tradeoff; over-tuned safety produces over-refusal, which is measured as a false-positive rate on benign prompts, not by eyeballing a few outputs.
Safety work begins with red teaming: adversarial probing for jailbreaks (prompts that bypass policy), prompt injection (especially indirect injection via retrieved documents or tool output), data exfiltration, and harmful-content generation. Distinguish jailbreak (attacker manipulates the model's own policy) from injection (attacker content overrides instructions). Guardrails - input/output classifiers, NeMo Guardrails-style dialogue rails, allow/deny lists, and tool-call validation - are runtime controls layered on top of alignment, not a substitute for it.
Hallucination control relies on grounding: retrieval with citations, abstention behavior, and faithfulness checks comparing answers to sources.
Bias evaluation uses counterfactual pairs (swap a protected attribute, compare outputs) and disaggregated metrics across subgroups, since aggregate scores hide harm. Privacy covers PII detection/redaction, memorization and extraction risk, and logging/retention policy.
Transparency artifacts include model cards, data cards, system cards, license terms (open weights is not the same as open source), evaluation disclosure, and human oversight with audit logs. Frameworks such as NIST AI RMF and the EU AI Act's risk tiers structure governance.
Traps
- Treating guardrails as equivalent to alignment training.
- Confusing jailbreak with prompt injection.
- Reporting only aggregate safety scores across subgroups.
- Assuming a model card or benchmark score certifies safety.
Worked example
A moderation classifier threshold trades precision against recall: lowering it catches more harm but blocks benign prompts. Measure both the harmful-catch rate and the benign false-positive rate on separate labeled sets before shipping.