CompTIA SecAI+ cheat sheet

NIST AI RMF has four core functions: GOVERN, MAP, MEASURE, MANAGE; GOVERN is continuous and cross-cutting, the other three form the iterative lifecycle.

EU AI Act risk tiers: unacceptable (banned), high-risk (conformity assessment, registration, logging), limited-risk (transparency duties), minimal-risk (no obligations).

ISO/IEC 42001 is a certifiable AI management system standard structured like ISO 27001: policy, roles, risk treatment, internal audit, continual improvement.

Risk = likelihood x impact; AI-specific impact dimensions include safety, fundamental rights, bias/discrimination, and model misuse.

Model cards document intended use, training data, evaluation metrics, and limitations; system cards cover the whole deployed system including guardrails.

Data provenance and lineage records (source, license, transformation, consent basis) are the evidence base for GDPR and copyright compliance claims.

A DPIA is required when processing is likely high-risk to individuals; AI profiling and large-scale sensitive-data inference typically trigger it.

Accountability requires a named risk owner per model plus documented approval gates; unowned models are the most common audit finding.

Post-market monitoring obligations require logging incidents, drift, and complaints, then feeding findings back into risk treatment.

Prompt injection is direct (user input) or indirect (poisoned retrieved documents, web pages, tool outputs); indirect is harder to filter because content arrives through trusted channels.

Jailbreaks target safety alignment (role-play, encoding, many-shot); prompt injection targets instruction hierarchy - mitigation differs.

Data poisoning corrupts training data; backdoor poisoning plants trigger-activated behavior that evades clean-validation testing.

Model extraction/stealing queries the API to clone behavior; rate limiting, query watermarking, and output perturbation raise cost.

Membership inference determines if a record was in training data; differential privacy during training is the primary defense.

Model inversion reconstructs representative training inputs from outputs, exposing sensitive attributes in overfit models.

Adversarial examples are small input perturbations causing misclassification; transferability means surrogate-model attacks can hit black-box targets.

Supply chain risk: unvetted model weights (pickle deserialization code execution), typosquatted packages, and poisoned public datasets.

Excessive agency: an LLM agent with write/execute permissions turns a prompt injection into real actions - scope tool credentials to least privilege.

Defense in depth for LLM I/O: input filtering, instruction/data separation, output validation, and action authorization are layered, not interchangeable.

Least-privilege tool design: each agent tool gets its own scoped credential; never reuse a broad service account across tools.

RAG security requires per-document access control enforced at retrieval time; a shared vector index leaks documents across tenants.

Vector stores can be poisoned - validate and sign ingested content, and tag chunks with source and trust level.

Differential privacy: epsilon controls privacy loss; smaller epsilon means stronger privacy and lower utility.

Encryption: at rest (AES-256), in transit (TLS 1.2+), and for inference isolation use confidential computing (TEEs) for sensitive prompts.

Model weight protection: encrypt artifacts at rest, restrict registry pull rights, and log every download as a sensitive event.

PII redaction before sending prompts to third-party model APIs prevents data leaving the trust boundary; DLP on egress is the enforcement point.

Guardrails should fail closed for high-impact actions and fail open only for low-risk read paths, decided per use case.

SOC triage with LLMs: model summarizes alerts and proposes severity, but analyst retains disposition authority for high-severity events.

Alert summarization quality depends on context windows; feed structured fields (source IP, hash, rule ID) rather than raw logs alone.

Hallucinated indicators are a core risk: any IOC, CVE, or hash the model produces must be verified against authoritative sources before action.

Automated response (block, isolate, disable account) needs human approval or a deterministic policy engine; never let the model call destructive APIs unguarded.

UEBA anomaly baselines drift; retrain schedules and drift alerts prevent false-positive storms after environment changes.

Model-assisted phishing detection uses text plus header/URL features; attackers evade by paraphrasing, so retrain on recent samples.

Adversarial evasion against ML detectors: attackers mutate artifacts until the classifier scores benign - ensemble and rule-based backstops reduce this.

Log the model's prompt, retrieved context, output, and confidence for every automated decision to preserve auditability.

Secure SDLC stages for AI: requirements/threat model, data curation, training, evaluation, deployment, monitoring, decommissioning.

Threat modeling for AI adds STRIDE-plus categories: prompt injection, data poisoning, model theft, and supply chain tampering.

MLOps gates: no model promotes to production without evaluation thresholds, bias checks, and a signed model card.

Model registry should be immutable per version with cryptographic hashes; deployments reference a digest, not a mutable tag.

Evaluation must include safety/red-team suites (jailbreak, injection, toxicity) alongside accuracy, not only benchmark scores.

Shadow deployment and canary rollout with rollback triggers limit blast radius of a bad model version.

Continuous monitoring covers input drift, output drift, safety-violation rate, and latency; define alert thresholds before launch.

Decommissioning requires revoking API keys, deleting weights and derived caches, and retaining audit records per retention policy.

CI/CD for AI must scan dependencies and model artifacts for known vulnerabilities and unsigned/unverified provenance.

NIST AI RMF core functions are GOVERN, MAP, MEASURE, MANAGE; GOVERN is continuous and cross-cutting while the other three iterate.

EU AI Act risk tiers: unacceptable (banned), high-risk (conformity assessment plus registration), limited-risk (transparency), minimal-risk (no obligations).

ISO/IEC 42001 defines a certifiable AI management system with policy, risk treatment, internal audit, and continual improvement.

Risk score = likelihood x impact, with AI impact including safety, fundamental rights, bias, and misuse potential.

Model cards record intended use, training data, evaluation metrics, and limitations for each model version.

Data provenance records (source, license, transformation, consent basis) supply evidence for GDPR and copyright compliance.

A DPIA is required when processing likely poses high risk to individuals, including AI profiling and sensitive inference.

Accountability requires a named risk owner per model and documented approval gates before deployment.

Post-market monitoring logs incidents, drift, and complaints and feeds findings back into risk treatment.

Prompt injection is direct (user input) or indirect (poisoned documents, web pages, tool output); indirect bypasses user-facing filters.

Jailbreaks target safety alignment; prompt injection targets the instruction hierarchy - mitigations differ.

Data poisoning corrupts training data; backdoor poisoning plants trigger-activated behavior that passes clean validation.

Model extraction queries the API to clone behavior; rate limiting, watermarking, and output perturbation raise attacker cost.

Membership inference tests whether a record was in training data; differential privacy during training is the main defense.

Model inversion reconstructs representative training inputs, exposing sensitive attributes in overfit models.

Adversarial examples are small perturbations causing misclassification; transferability enables black-box attacks via surrogate models.

Supply chain risk includes pickle-based weight deserialization, typosquatted packages, and poisoned public datasets.

Excessive agency turns prompt injection into real actions; scope tool credentials to least privilege.

Layer LLM defenses: input filtering, instruction/data separation, output validation, and action authorization are complementary controls.

Each agent tool gets its own scoped credential; never share a broad service account across tools.

RAG access control must be enforced at retrieval time per document; a shared vector index leaks across tenants.

Validate and sign ingested content and tag chunks with source and trust level to resist vector-store poisoning.

Differential privacy uses epsilon: smaller epsilon means stronger privacy and lower utility.

Use AES-256 at rest, TLS 1.2+ in transit, and confidential computing (TEEs) for sensitive inference isolation.

Protect model weights by encrypting artifacts, restricting registry pulls, and logging every download.

Redact PII before sending prompts to third-party APIs; DLP on egress is the enforcement point.

Guardrails fail closed for high-impact actions and may fail open only for low-risk read paths.

LLM triage summarizes alerts and proposes severity; analysts retain disposition authority for high-severity events.

Feed structured fields (source IP, hash, rule ID) rather than raw logs to improve summarization quality.

Verify every model-produced IOC, CVE, or hash against authoritative sources before acting; hallucination is a core risk.

Automated response (block, isolate, disable) requires human approval or a deterministic policy engine.

UEBA baselines drift; scheduled retraining and drift alerts prevent false-positive storms.

Phishing classifiers combine text and header/URL features; paraphrase evasion requires retraining on recent samples.

Attackers mutate artifacts until ML detectors score benign; ensembles and rule-based backstops reduce evasion.

Log prompt, retrieved context, output, and confidence for every automated decision to preserve auditability.

AI secure SDLC stages: requirements and threat model, data curation, training, evaluation, deployment, monitoring, decommissioning.

AI threat modeling adds prompt injection, data poisoning, model theft, and supply chain tampering to standard categories.

MLOps gates block promotion unless evaluation thresholds, bias checks, and a signed model card are satisfied.

Model registry entries are immutable per version with cryptographic hashes; deployments reference a digest, not a mutable tag.

Evaluation includes red-team safety suites (jailbreak, injection, toxicity) alongside accuracy benchmarks.

Shadow deployment and canary rollout with rollback triggers limit the blast radius of a bad model version.

Continuous monitoring tracks input drift, output drift, safety-violation rate, and latency against pre-set thresholds.

Decommissioning revokes API keys, deletes weights and derived caches, and retains audit records per retention policy.

CI/CD scans dependencies and model artifacts for known vulnerabilities and unverified provenance.