CAIS flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

NIST AI RMF core functions

GOVERN, MAP, MEASURE and MANAGE. GOVERN is cross-cutting and sets accountability, culture and risk tolerance; the other three form the iterative lifecycle used to identify, assess and treat AI risk.

EU AI Act risk tiers

Four tiers: unacceptable (prohibited), high-risk (conformity assessment, technical documentation, human oversight, registration), limited-risk (transparency duties such as disclosing AI interaction), and minimal-risk (no added obligations).

ISO/IEC 42001

Certifiable international standard for an AI management system. It follows Plan-Do-Check-Act and specifies organisational controls for AI risk, impact assessment, data quality, competence and third-party relationships.

AI system roles under EU AI Act

Provider develops or places the system on the market and bears conformity duties; deployer uses it professionally; authorised representative acts for non-EU providers; importer and distributor move it within the EU market.

Seven AI RMF trustworthiness traits

Valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. MEASURE tests systems against these characteristics.

Residual versus inherent risk

Inherent risk is exposure before controls; residual risk remains after mitigation. Acceptance decisions compare residual risk against the documented risk appetite and tolerance thresholds, not against the inherent rating.

Four risk treatment options

Mitigate by applying a control, transfer through insurance or contract, avoid by not deploying the system, or accept with documented sign-off by an accountable owner acting within approved risk appetite.

NIST AI RMF MAP function

Establishes context: intended use, affected stakeholders, assumptions, dependencies and the likelihood and impact of harms. Its output feeds MEASURE and MANAGE and is revisited when context changes.

Evasion attack

An adversary crafts inputs at inference time to cause misclassification, for example FGSM or PGD perturbations. No training access is required, and effectiveness often transfers across models, making it a live production threat.

Backdoor poisoning

Training-time manipulation that binds a trigger pattern to a chosen output, so the model behaves normally until the trigger appears. Clean-label variants keep labels correct and evade manual review.

Membership inference attack

Determines whether a specific record was in the training set by exploiting overfitting in confidence scores. It is a privacy attack; mitigations include regularisation, differential privacy and confidence output truncation.

Indirect prompt injection

Malicious instructions hidden in content the model retrieves or reads, such as web pages, documents or tool output, rather than typed by the user. Retrieved content must be treated as untrusted data, never instructions.

Unsafe model deserialization

Loading a pickle-format model executes arbitrary Python during unpickling, giving remote code execution. Prefer safetensors or ONNX and scan third-party model artefacts before loading them in production.

Vector database exposure

Embeddings and stored chunks often contain raw personal data or secrets and can be approximately inverted. Treat the vector store as a sensitive repository with access control, encryption, tenant isolation and retention limits.

Dataset provenance record

Captures source, collection date, licence, consent basis and transformation history plus a cryptographic hash per version. It enables poisoning investigation, reproducibility and deletion when a source is withdrawn.

Data minimization for training

Collect and retain only data needed for the stated model purpose. This reduces breach impact, poisoning surface and regulatory exposure, and simplifies satisfying erasure requests against trained models.

Differential privacy epsilon

A privacy budget parameter: lower epsilon means stronger privacy and more added noise. The budget is consumed across queries, so repeated releases of the same data compound privacy loss.

Machine unlearning

Techniques that remove a data subject's influence from a trained model without full retraining, supporting erasure rights. Verify with membership inference tests; exact removal usually requires retraining or influence-function methods.

Demographic parity

A fairness criterion requiring equal positive prediction rates across groups. It ignores base rates, so it can conflict with equalized odds; choosing the metric is a policy decision, not a purely technical one.

NIST Privacy Framework functions

Identify-P, Govern-P, Control-P, Communicate-P and Protect-P. Govern-P is cross-cutting; the framework is voluntary and maps to GDPR obligations and ISO/IEC 27701 controls for a privacy programme.

Least agency for LLM agents

Grant an agent only the tools, scopes and data its task requires, with human approval for irreversible actions. This limits blast radius when prompt injection or hallucination drives a harmful tool call.

AI bill of materials

An inventory of components, pretrained weights, datasets, licences and dependencies in an AI system. It enables vulnerability triage and supply-chain attestation, analogous to an SBOM in conventional software.

Improper output handling

OWASP LLM05: downstream systems render, execute or trust model output without validation, turning injected content into cross-site scripting, SSRF or command injection. Treat all model output as untrusted user input.

STRIDE for AI threat modeling

Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege. Apply per component - data intake, training, model store, inference API, agent tools - and add AI-specific cases such as poisoning.

Concept drift

Change over time in the relationship between inputs and target, degrading model accuracy. Detected by monitoring prediction quality against delayed ground truth, which triggers retraining or retirement decisions.

Model registry controls

A central store for versioned artefacts with access control, immutable versions, hash pinning and approval status. Production deployments should reference a signed, approved digest, never a mutable tag like latest.

Shadow deployment

A new model runs on live traffic in parallel without serving responses, and its outputs are compared with the incumbent. This validates behaviour and latency before canary or blue-green promotion.

Reproducible training build

Pinned dependencies, containerised environments, seeded randomness and versioned data snapshots. It enables incident investigation, rollback to a known-good model, and detection of tampering in the training pipeline.

AI incident containment actions

Disable agent tools or API keys, roll back to the last approved model version, block the offending input pattern, and preserve logs, prompts, dataset versions and model hashes before remediation destroys evidence.

AI red teaming

Adversarial testing of safety, security and misuse using prompt injection, jailbreaks, data extraction and poisoning attempts. It differs from penetration testing by targeting model behaviour and guardrail bypass, not just infrastructure.

Mean time to detect

Average elapsed time between an AI security event occurring and its detection. It is a core assurance KPI, reduced by anomaly baselines over token volume, refusal rate, tool calls and data egress.

Assurance evidence for AI

Audit artefacts: risk assessments, test and red-team reports, control mappings, monitoring metrics and change records. Evidence must be current, traceable to a specific system version and retained for the audit period.