CAICSO flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

NIST AI RMF GOVERN function

Cross-cutting function that establishes accountability, policies, and culture for AI risk management and applies across MAP, MEASURE, and MANAGE activities.

ISO/IEC 42001

Certifiable international standard specifying requirements for an AI management system, including AI-specific controls in Annex A and continual improvement requirements.

EU AI Act prohibited practices

Uses banned outright, including social scoring, manipulative techniques exploiting vulnerabilities, and certain real-time remote biometric identification in public spaces.

Model card

Short document describing a model's intended use, training data, performance metrics, and known limitations to support informed downstream use.

Three lines model

Governance structure where operational teams own risk, risk and compliance functions provide oversight, and internal audit gives independent assurance.

Inherent versus residual risk

Inherent risk is scored before controls are applied; residual risk is scored after controls, and acceptance requires a named accountable owner.

Allocative versus representational harm

Allocative harm is unequal distribution of resources or opportunities; representational harm is stereotyping or degrading portrayal of groups.

Disparate impact metrics

Group-level fairness measures such as demographic parity difference and equalized odds that reveal bias hidden by aggregate accuracy figures.

AI threat modeling extensions

STRIDE augmented with AI-specific threats: adversarial input, data poisoning, model extraction, prompt injection, and model supply-chain compromise.

Indirect prompt injection

Malicious instructions embedded in retrieved documents, web pages, or tool outputs that the model processes as trusted context and may execute.

Data poisoning attack

Corruption of training data to implant backdoors or bias model behavior; clean-label variants keep labels correct while perturbing features.

Backdoor trigger attack

Model behaves normally except when a specific trigger pattern appears, producing attacker-chosen output while validation accuracy remains high.

Model extraction attack

Repeated queries to a model to approximate its function or decision boundary, enabling theft of intellectual property or surrogate attack development.

Membership inference attack

Determining whether a specific record was part of a model's training data, typically enabled by overfitting and overconfident output scores.

Evasion attack

Inference-time input manipulation designed to cause misclassification or policy bypass without any access to training data or model weights.

Jailbreak versus prompt injection

Jailbreaks bypass safety alignment to elicit prohibited content; prompt injection hijacks the instruction hierarchy to redirect system behavior.

Retrieval access control enforcement

RAG systems must filter retrieved chunks by the requesting user's entitlements at query time so context never exceeds authorized access.

Model artifact integrity verification

Checking hashes or signatures of weights, tokenizers, and configs before loading, and avoiding unsafe deserialization formats such as pickle.

Agent least privilege

Each model or agent runs under a scoped identity with explicit tool and API allowlists, rate limits, and no standing broad credentials.

Output guardrails

Post-generation controls including schema validation, allowlisted formats, secret and PII redaction, and content policy classifiers before delivery.

Differential privacy epsilon

Parameter quantifying privacy loss; smaller epsilon gives stronger privacy guarantees and generally reduces model utility and accuracy.

Training data memorization

Model reproduction of rare or duplicated training sequences; risk rises with capacity and duplication and is tested using canary strings.

Prompt and log retention

Prompts and responses often contain PII, so they require redaction before storage and explicit retention windows to limit exposure.

Machine unlearning

Technique to remove influence of specific training records from a deployed model when full retraining is infeasible after a deletion request.

ML supply chain pinning

Locking dependency versions and verifying hashes of base models and libraries so training and inference use only approved artifacts.

Pre-release evaluation gate

Mandatory checkpoint requiring safety, red-team, bias, performance, and artifact integrity results before a model version can deploy.

Canary model deployment

Releasing a new model version to a small traffic slice to detect regressions and attacks before full rollout, with rollback ready.

Model retirement controls

Revoking credentials, archiving artifacts for audit, notifying downstream consumers, and removing endpoints when a model is decommissioned.

Model drift detection

Monitoring shifts in input distributions and output quality against deployment baselines to trigger retraining or rollback decisions.

AI incident containment actions

Disabling tools, tightening guardrails, rolling back model or prompt versions, or switching to restricted safe mode to stop harm.

AI incident severity factors

Affected population size, reversibility of harm, autonomy of the decision, and regulatory notification obligations determine incident severity.

AI review board

Cross-functional body with security, legal, privacy, data science, and business owners that approves high-risk AI use cases and exceptions.