ISTQB CT-AI flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

AI-based system

A system that perceives its environment and acts to achieve defined objectives, using data-driven or rule-based reasoning, and that may change its behaviour through learning.

Narrow AI

An AI system designed for one bounded task, such as image classification or recommendation, without general reasoning ability transferable to unrelated domains.

Supervised learning

A machine learning paradigm that trains a model on labelled input-output pairs so the model can predict the label of previously unseen inputs.

Reinforcement learning

A paradigm in which an agent learns a policy by interacting with an environment and maximising cumulative reward through trial, error and feedback signals.

Oracle problem in AI testing

The difficulty of determining the expected output for a test input, because correct model behaviour is probabilistic, subjective or only partially specified in requirements.

ISO/IEC 25059

An extension of the ISO/IEC 25010 SQuaRE product quality model that defines quality characteristics and sub-characteristics specifically applicable to AI systems.

Flexibility quality characteristic

The ISO/IEC 25059 characteristic that replaces portability, covering adaptability, scalability, installability and replaceability of an AI system across changing conditions.

Safety quality characteristic

The ISO/IEC 25059 characteristic describing freedom from unacceptable risk of harm to people, property or the environment arising from operation of an AI system.

AI security threats

Attacks such as data poisoning, adversarial evasion, model extraction and membership inference that compromise the integrity, confidentiality or availability of AI systems.

Robustness testing

Evaluating whether a model keeps acceptable performance under perturbed inputs, noisy or missing data, distribution shift and adversarial conditions.

CRISP-DM phases

Business understanding, data understanding, data preparation, modelling, evaluation and deployment; the lifecycle is iterative rather than strictly sequential.

Purpose of a validation set

A held-out data subset used to tune hyperparameters and select models, keeping the test set untouched for a final unbiased estimate of quality.

k-fold cross-validation

Splitting data into k folds, training k times with each fold used once for validation, then averaging the results to reduce evaluation variance.

Data leakage

Contamination of training data with information about the target or the test set, producing optimistic evaluation results that do not hold in production.

Concept drift

A change over time in the statistical relationship between input features and the target variable, degrading model performance until the model is retrained.

Training/test split independence

Test data must be disjoint from training data, including duplicates and near-duplicates, otherwise the reported model quality is inflated and misleading.

Inter-annotator agreement

A measure, often Cohen's kappa, of how consistently several labellers assign the same labels; low agreement signals ambiguous data or poor labelling guidelines.

Class imbalance impact

When one class dominates, accuracy misleads; per-class precision, recall and balanced metrics are needed to expose biased predictions.

Representativeness of test data

The degree to which input data reflects the operational population, including subgroups, edge cases and rare conditions the model will encounter in use.

Metamorphic testing

A technique that derives expected relations between outputs of related inputs, for example rotating an image should not change its classification, avoiding fixed oracles.

Confusion matrix

A table of predicted versus actual classes that yields accuracy, precision, recall and F1, and shows which classes are systematically misclassified.

Precision versus recall

Precision is the share of positive predictions that are correct; recall is the share of actual positives that are found; they trade off through the decision threshold.

Adversarial example

An input deliberately perturbed with small, often imperceptible, changes that causes a trained model to produce an incorrect or unsafe output.

Back-to-back testing

Running the same inputs through two or more model versions or implementations and comparing their outputs to detect regressions or unexpected discrepancies.

Model regression testing

Re-running a suite of model-level tests after data, code or hyperparameter changes to confirm that quality has not degraded before release.

Reproducibility in ML

The ability to obtain the same results from a training run, requiring fixed random seeds, pinned dependencies, versioned data and recorded configurations.

Shadow deployment

Releasing a new model to receive production traffic in parallel with the incumbent without acting on its outputs, so behaviour can be compared safely.

Canary release

Gradually routing a small percentage of live traffic to a new model, monitoring quality metrics and rolling back if performance degrades.

Hallucination

A generative model output that is fluent and plausible but factually unsupported or fabricated, typically detected by grounding checks against trusted sources.

LLM-as-a-judge

Using a language model to score another model's outputs against a rubric; the judge must itself be validated for bias and agreement with human raters.

Prompt injection

An attack in which adversarial text in the user input or retrieved content overrides the model's instructions, causing unsafe or unintended actions.

Groundedness in RAG

The degree to which a generated answer is supported by the retrieved context, used as a primary check for hallucination in retrieval-augmented systems.