CAICSO key terms

A short document shipped with a model that states its intended use, training data characteristics, evaluation results, and known limitations, so downstream users can judge whether the model fits their context.

A maintained register of every AI system in an organization, recording owner, purpose, data sources, model dependencies, and deployment environment, forming the basis for governance, risk, and incident scoping.

Assigning AI systems to risk bands based on factors such as decision impact on people, data sensitivity, autonomy, and reversibility, so that controls and review depth are proportionate to potential harm.

A designed control in which a competent person can monitor, intervene in, or override an AI system's outputs, with defined authority, timing, and escalation paths rather than nominal review.

A voluntary US framework organizing AI risk work into govern, map, measure, and manage functions, used to structure programs, evidence, and continuous improvement without prescribing specific technologies.

An international management system standard for AI that specifies requirements for establishing, operating, and continually improving an AI management system, including policies, roles, risk treatment, and internal audit.

A structured review of how a proposed or existing AI system affects individuals, groups, and the organization, covering benefits, harms, affected stakeholders, mitigations, and residual risk acceptance.

Systematically enumerating adversaries, entry points, assets, and attack paths specific to an AI system, including data pipelines, model artifacts, prompts, and integrations, to prioritize security controls before deployment.

The risk that remains after planned controls are applied, which must be documented and formally accepted by an accountable owner rather than assumed to be eliminated.

The amount and type of risk leadership is willing to accept in pursuit of objectives, expressed in thresholds that guide decisions such as whether to deploy, restrict, or retire an AI system.

The full set of points where an adversary could interact with or influence a system, including APIs, user prompts, retrieved documents, training pipelines, model files, plugins, and third-party dependencies.

Layering independent controls such as input filtering, output validation, least-privilege tool access, and monitoring so that failure of any single control does not result in full compromise.

Granting each model, agent, and pipeline only the data, tools, and permissions needed for its task, limiting blast radius when a model is manipulated or behaves unexpectedly.

Inspecting or constraining model responses before they reach users or downstream systems, for example by schema checks, allowlists, or content filters, to block unsafe or malformed actions.

Runtime controls that constrain model behavior, such as policy prompts, classifiers, tool allowlists, and rate limits, intended to keep outputs within approved boundaries during operation.

An attack in which adversarial instructions are placed in model input, including retrieved or web content, causing the model to follow attacker intent instead of the operator's instructions.

A variant where malicious instructions arrive through content the model processes, such as documents, emails, or web pages, rather than from the user, often enabling data exfiltration or unauthorized tool use.

Crafting input that bypasses a model's safety alignment or policy restrictions, typically through role-play, encoding, or multi-turn manipulation, so the model produces prohibited content or actions.

Deliberately corrupting or manipulating training or fine-tuning data so the resulting model misbehaves on chosen inputs, degrades generally, or carries a hidden trigger.

A pattern or phrase inserted during malicious training that causes a model to produce attacker-chosen behavior only when the trigger appears, remaining inconspicuous on normal inputs.

Querying a deployed model systematically to reconstruct an approximate copy or steal its behavior, enabling the attacker to avoid fees, bypass controls, or probe for weaknesses offline.

An attack that determines whether a specific record was part of a model's training data by exploiting differences in model confidence, which can reveal sensitive information about individuals.

A formal technique that adds calibrated noise so that outputs reveal little about any single training record, providing a measurable privacy guarantee controlled by a privacy budget parameter.

Collecting and retaining only the data actually needed for a defined purpose, reducing the volume of sensitive information exposed to training, logging, and retrieval systems.

The documented origin, lineage, and transformation history of data used to train or ground a model, supporting licensing checks, poisoning detection, and regulatory accountability.

Detecting and removing or masking personal identifiers before text is sent to a model or stored in logs, limiting exposure through third-party inference providers and prompt logging.

Unintended disclosure of confidential or personal information through model outputs, logs, retrieval caches, or agent tool calls, often caused by over-broad context or inadequate output controls.

Exposure introduced by third-party models, datasets, libraries, and pretrained weights whose provenance, integrity, or licensing cannot be fully verified by the consuming organization.

Cryptographically signing model weights and related artifacts so that deployment pipelines can verify integrity and origin, detecting tampering or substitution between training and serving.

A controlled repository tracking model versions, training metadata, approvals, and deployment status, providing traceability and a gate for promoting models into production.

Running a new model version alongside the current one on live traffic without affecting users, to compare behavior and detect regressions before full rollout.

Degradation of model performance as real-world data diverges from training conditions, requiring monitoring, retraining triggers, and periodic revalidation of security assumptions.

Continuously observing model inputs, outputs, and actions for anomalies such as unusual refusal rates, unexpected tool calls, or spikes in sensitive content, to detect attacks and failures early.

An event where an AI system causes or contributes to harm, security compromise, or policy violation, triggering defined response, containment, notification, and lessons-learned processes.

A predefined scale mapping AI incident characteristics such as data exposure, safety impact, and scope to response urgency, escalation level, and notification obligations.

A tested mechanism that rapidly disables or rolls back a model, agent, or tool integration during an incident, with clear authority over who may activate it.

A condition where a model or agent can take actions beyond what its task requires, such as sending data externally or modifying records, amplifying the impact of manipulation or error.

The human tendency to over-trust automated outputs and reduce independent scrutiny, which weakens human oversight controls and increases the chance that model errors go unchallenged.

Role-specific education for staff on AI-specific threats such as prompt injection, data leakage, and unsafe tool use, including acceptable use rules and how to report suspected incidents.

Restricting query volume or concurrency per user or key to reduce abuse, slow model extraction and probing attacks, and contain cost overruns from automated misuse.