EXIN AI Security Professional key terms

An attack in which untrusted text fed to a model, such as a document or web page, contains instructions that override the developer's intended behaviour and cause the model to act on the attacker's goals.

A prompt injection delivered through content the model retrieves or processes rather than through the user's own message, for example a poisoned email, web page, or retrieved document that carries hidden instructions for the model to follow.

A technique that persuades a model to bypass its own safety alignment, typically through role-play, hypothetical framing, or encoded requests, so that it produces content its operator intended to refuse.

An input deliberately perturbed, often imperceptibly to humans, so that a model misclassifies it or behaves incorrectly; a classic integrity attack on machine-learning classifiers and perception systems.

An attack performed at inference time that crafts inputs to slip past a deployed model's decision boundary, for example a malware sample modified to be classified as benign, without altering the model itself.

Manipulation of training or fine-tuning data so that a model learns attacker-chosen behaviour, such as a hidden trigger or a biased association, which then persists in the deployed model.

A specific pattern secretly implanted during training that causes a model to produce an attacker-desired output whenever the trigger appears, while behaving normally on ordinary inputs.

Repeatedly querying a deployed model's API to reconstruct a functionally similar copy or to infer its parameters, enabling the attacker to steal intellectual property and to study the model for later attacks.

An attack that uses a model's outputs to reconstruct sensitive information about its training data, such as recovering recognisable images of individuals from a face-recognition model's confidence scores.

An attack that determines whether a specific record was part of a model's training set, which can reveal sensitive facts about individuals and indicates overfitting or insufficient privacy protection.

A formal privacy guarantee that adds calibrated noise so that outputs change only marginally whether or not any single individual's data is included, bounding what an observer can learn about that person.

An architecture in which a model retrieves documents from an external store and conditions its answer on them, improving factual grounding while widening the attack surface through untrusted retrieved content.

A store that indexes embeddings so that semantically similar items can be found quickly; in AI systems it holds the retrievable knowledge base and therefore becomes a target for poisoning and unauthorised access.

A dense numeric representation of text, images, or other data in a high-dimensional space where semantic similarity corresponds to closeness; embeddings can leak information and must be protected like the source data.

A control placed around a model, such as input filtering, output moderation, or policy checks, that detects and blocks unsafe or disallowed content before it reaches users or downstream systems.

Manipulating an agent so that its permitted tools, such as code execution, email, or payment APIs, are invoked with attacker-chosen parameters, turning legitimate capabilities into a path for harm.

AI systems that plan and take sequences of actions through tools and APIs with limited human involvement, which raises security concerns about autonomy, scope of permissions, and accountability for actions taken.

The principle of granting a user, service, or AI agent only the access needed for its current task, limiting the blast radius if the component is compromised or manipulated into misusing its permissions.

Structured adversarial testing in which specialists deliberately try to make an AI system fail, misuse tools, or produce harmful output, documenting findings so that controls and mitigations can be improved.

The chain of datasets, pretrained weights, libraries, and hosting services that produce a deployed model; each link can introduce tampering, licences, or vulnerabilities that the deploying organisation inherits.

A machine-readable inventory of the components and dependencies in a software product, increasingly extended to AI models and datasets so that organisations can trace provenance and respond to disclosed vulnerabilities.

A short document describing a model's intended use, training data, performance, and known limitations, which supports informed deployment decisions and transparent communication of residual risk.

The principle that personal data collected and used should be adequate, relevant, and limited to what is necessary for the stated purpose, reducing privacy exposure in training and inference pipelines.

The rule that personal data collected for one stated purpose must not be reused for an incompatible purpose without a lawful basis, a constraint that often complicates repurposing datasets for model training.

The EU General Data Protection Regulation, which governs processing of personal data and grants rights such as access, erasure, and explanation, all of which interact with how AI systems are trained and operated.

A data subject's right to have personal data deleted, which creates technical challenges for trained models because removing one person's influence from learned parameters is difficult once training is complete.

Techniques for removing the influence of specific training records from a trained model without full retraining, used to support erasure requests and to purge poisoned or unlawfully obtained data.

The European Union regulation establishing harmonised rules for placing AI systems on the market, structured around risk tiers, with prohibitions, transparency duties, and conformity obligations for higher-risk systems.

The regulatory logic of the EU AI Act, under which obligations scale with the level of risk a system poses, from minimal obligations for most systems to strict requirements for high-risk and prohibited uses.

An AI use that the EU AI Act bans outright, such as certain social scoring, manipulative techniques exploiting vulnerabilities, and untargeted scraping of facial images to build recognition databases.

An AI system classified under the EU AI Act as posing significant risk to health, safety, or fundamental rights, typically because it is a safety component or is used in listed areas such as employment or education.

Under the EU AI Act, a model trained on broad data that can perform a wide range of tasks and be integrated into many downstream systems, subject to transparency duties and extra obligations when systemic risk is present.

A risk designation for general-purpose AI models with high-impact capabilities, triggering additional duties such as model evaluation, adversarial testing, incident reporting, and cybersecurity protection for the model and its weights.

The documented procedure by which a provider demonstrates that a high-risk AI system meets the EU AI Act's requirements before placing it on the market, supported by technical documentation and quality management evidence.

The marking affixed to a product to indicate conformity with applicable EU requirements; for high-risk AI systems it signals that the provider has completed the required assessment and accepts legal responsibility.

The record a provider must prepare describing a system's design, data, training, testing, and risk controls, enabling authorities to assess compliance and supporting traceability of security decisions over time.

A duty to inform users or affected persons that they are interacting with AI, that content is synthetic, or that certain risks exist, so that people can make informed decisions about the system's outputs.

A designed capability for people to monitor, intervene in, or override an AI system, intended to prevent harm and to keep meaningful human control over consequential decisions, especially in high-risk settings.

A continuous, documented process required for high-risk AI systems that identifies, evaluates, and mitigates risks across the lifecycle, including security threats, and is updated as evidence changes.

The ongoing collection and analysis of data on a deployed AI system's performance and incidents after release, used to detect degradation or new risks and to trigger corrective action.

The gradual decline in a deployed model's accuracy or safety as real-world data diverges from its training distribution, a reason why monitoring and periodic re-evaluation are security and compliance requirements.

An analysis, required in certain EU AI Act cases, of how a high-risk system may affect rights such as privacy, non-discrimination, and dignity, and of the measures taken to address those impacts.

The obligation to notify authorities or affected parties when an AI system causes or contributes to a serious incident, supporting rapid containment and shared learning about emerging failure modes.

The policies, roles, and oversight mechanisms an organisation uses to direct and control AI development and use, aligning technical teams, legal duties, and risk appetite into accountable decision-making.

The assignment of clear responsibility for AI systems and their outcomes to named roles, including the ability to demonstrate that controls were designed and operated as intended.

The exposure an organisation faces from vendors, model providers, and open-source components it relies on, managed through due diligence, contractual security terms, and ongoing assurance of external dependencies.

Systematic measurement of a model's capability, accuracy, bias, and safety behaviour on defined test sets and scenarios, producing evidence that informs release decisions and regulatory documentation.

The mechanisms that decide which identities may read, modify, or invoke a resource, essential for protecting model weights, training data, and retrieval stores from unauthorised use or tampering.

Under the EU AI Act, a machine-based system designed to operate with varying autonomy, inferring from input how to produce outputs that can influence physical or virtual environments and decisions.