EXIN AI Security Professional cheat sheet

Threat taxonomy for ML systems: evasion (perturbed inputs at inference), poisoning (manipulated training data), model inversion and membership inference (privacy extraction), model theft/extraction (query-based copying), and prompt injection (direct or indirect) for LLM-based systems.

Adversarial robustness metrics: fast gradient sign method (FGSM) perturbs input by epsilon * sign(gradient of loss); projected gradient descent (PGD) is the iterative multi-step variant; robustness is reported as accuracy under attack at a fixed perturbation budget (L-infinity norm epsilon).

Poisoning defence decision rule: if attacker controls less than roughly 1% of training labels, backdoor success can still be high, so volume thresholds alone are insufficient; use data provenance tracking, outlier/activation-clustering detection, and influence-function auditing instead.

Differential privacy: epsilon is the privacy budget; smaller epsilon means stronger privacy and more noise; composition means k queries with epsilon each accumulate toward k*epsilon under basic composition, and (epsilon, delta)-DP allows a delta failure probability.

Membership inference risk increases with overfitting: the gap between training and test accuracy is a practical proxy; mitigations are regularization, early stopping, output confidence truncation, and DP training (e.g. DP-SGD with per-example gradient clipping).

Model extraction defence: rate-limit and monitor query patterns, return rounded or top-k-only probabilities, add watermarking or fingerprinting to detect copied models, and require authenticated API access with per-tenant quotas.

Prompt injection controls: treat retrieved documents and tool outputs as untrusted data, separate instruction and data channels, apply least-privilege tool scopes, require human confirmation for high-impact actions, and validate model output before execution.

ML supply chain: ML models are serialized code (pickle) and can execute on load, so prefer safetensors or ONNX, verify hashes and signatures, scan model artifacts, and pin dataset and base-model versions in an inventory (AI bill of materials).

AI governance control mapping: ISO/IEC 27001 Annex A covers information security management, ISO/IEC 42001 specifies an AI management system, and NIST AI RMF organizes work into Govern, Map, Measure, Manage functions; these are complementary, not substitutes.

EU AI Act risk tiers: prohibited practices (Article 5), high-risk systems (Article 6 and Annex III), limited-risk systems with transparency duties (Article 50), and minimal-risk systems with no additional obligations; GPAI models carry separate obligations under Articles 51-56.

Prohibited practices (Article 5) include social scoring by public authorities, subliminal or manipulative techniques causing significant harm, exploitation of vulnerabilities, untargeted facial scraping for facial recognition databases, emotion recognition in workplaces and schools, biometric categorization of sensitive attributes, and most real-time remote biometric identification in public spaces for law enforcement.

High-risk classification triggers: a system is high-risk if it is a safety component of a product under listed Union harmonisation legislation, or falls in Annex III areas such as biometrics, critical infrastructure, education, employment, essential services, law enforcement, migration, and justice. Article 6(3) provides a narrow exception when the system performs a purely preparatory or narrow procedural task.

High-risk provider obligations: risk management system across the lifecycle, data governance, technical documentation, logging and record-keeping, transparency and instructions for use, human oversight, accuracy/robustness/cybersecurity, quality management system, conformity assessment, CE marking, and EU database registration.

GPAI obligations: all providers must supply technical documentation, information to downstream providers, a copyright compliance policy, and a training data summary; models with systemic risk (training compute above 10^25 FLOP) additionally require model evaluation, adversarial testing, incident reporting, and cybersecurity protection.

Timeline: the AI Act entered into force on 1 August 2024; prohibitions applied from 2 February 2025; GPAI obligations from 2 August 2025; most high-risk obligations from 2 August 2026; and high-risk systems embedded in regulated products from 2 August 2027.

Roles and responsibilities: providers place systems on the market and bear conformity duties; deployers use systems under their authority and must conduct a fundamental rights impact assessment in specified cases; importers and distributors verify compliance; authorised representatives act for non-EU providers.

Transparency duties (Article 50): disclose AI interaction, mark synthetic content machine-readably, label deepfakes, and inform people exposed to emotion recognition or biometric categorisation; general-purpose AI output must be detectable as AI-generated.

Penalties: up to EUR 35 million or 7% of global annual turnover for prohibited practices, EUR 15 million or 3% for most other breaches, and EUR 7.5 million or 1% for supplying incorrect information; the higher of the fixed amount and the percentage applies.