CAIP (AIP-210) flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Bias-variance tradeoff

Total expected error equals squared bias plus variance plus irreducible noise. Increasing model complexity lowers bias but raises variance; regularization and more data reduce variance.

Supervised learning

Training on labeled pairs of inputs and target outputs so the model learns a mapping that generalizes to unseen inputs. Examples: classification and regression.

L1 vs L2 regularization

L1 adds the absolute value of weights and produces sparse solutions for feature selection. L2 adds squared weights and shrinks coefficients smoothly without zeroing them.

Precision-recall tradeoff

Lowering the classification threshold raises recall and lowers precision. Choose the operating point from the relative business cost of false negatives versus false positives.

ROC-AUC

Area under the true-positive versus false-positive rate curve. It measures ranking quality across thresholds and ranges from 0.5 (random) to 1.0 (perfect).

Overfitting

A model fits training noise rather than the underlying signal, showing low training error but high validation error. Remedies include regularization, more data, and simpler models.

Reinforcement learning

An agent learns a policy by taking actions in an environment and maximizing cumulative reward. Key concepts are state, action, reward, and exploration versus exploitation.

Amazon SageMaker

Managed AWS service covering the ML lifecycle: data labeling, notebooks, training jobs, automatic tuning, model registry, endpoints, and pipelines for orchestration.

Amazon Comprehend

Managed natural language processing service providing entity detection, sentiment analysis, key-phrase extraction, language detection, and custom text classification without training infrastructure.

Confusion matrix

A table of true positives, false positives, false negatives, and true negatives. Accuracy, precision, recall, and F1 are all derived from these four counts.

Cross-validation

Splitting data into k folds, training on k-1 and validating on the held-out fold, then averaging. It gives a more stable performance estimate than a single split.

Data leakage

Information from the target or future data contaminates training features, producing unrealistically high validation scores that fail in production. Fit preprocessing inside training folds only.

Train/validation/test split

Training fits parameters, validation guides model selection and hyperparameters, and test provides one final unbiased estimate. The test set must never inform tuning decisions.

Class imbalance

When one class dominates, accuracy misleads. Use resampling, class weights, threshold tuning, and metrics such as PR-AUC or recall instead of raw accuracy.

Baseline model

A simple reference such as predicting the majority class or mean. Any complex model must beat it measurably to justify added cost and complexity.

SageMaker Automatic Model Tuning

A hyperparameter tuning job that searches specified ranges, using Bayesian optimization by default, to maximize or minimize a chosen objective metric.

Feature engineering

Creating or transforming input variables - encoding categoricals, normalizing numerics, deriving ratios and time features - to make patterns easier for the model to learn.

Stratified k-fold

Cross-validation that preserves the class proportions in each fold. It is preferred for imbalanced classification because plain k-fold can produce folds missing a class.

Time series splitting

Splits must be chronological: train on earlier periods and validate on later ones. Random shuffling leaks future information and overstates performance.

One-hot encoding

Representing a categorical feature as binary columns, one per category. It avoids implied ordering but creates high dimensionality for high-cardinality features.

ML objective vs business objective

The business objective is the real outcome (revenue, churn); the ML objective is the loss optimized during training. The evaluation metric must bridge the two.

SageMaker Pipelines

A managed workflow service that orchestrates preprocessing, training, evaluation, and conditional model registration steps, supporting reproducibility and automated retraining.

Real-time SageMaker endpoint

A persistently hosted endpoint for low-latency synchronous inference. It incurs cost while running, so it suits steady, latency-sensitive production traffic.

Serverless inference

An endpoint option that scales to zero when idle and bills per request, suited to intermittent or unpredictable traffic with tolerance for cold-start latency.

Batch transform

An offline SageMaker job that scores large datasets in bulk without hosting a persistent endpoint, ideal for periodic scoring where latency does not matter.

Canary deployment

Releasing a new model to a small fraction of traffic first, monitoring metrics, then gradually increasing exposure or rolling back if errors rise.

SageMaker Model Monitor

Detects drift on deployed models: data drift, model-quality drift, bias drift, and feature-attribution drift, emitting CloudWatch alerts when baselines are exceeded.

SageMaker Model Registry

A versioned catalog of trained models with approval status and metadata, used to govern promotion of models from development to production.

Least privilege IAM role

Granting the model or pipeline only the permissions it needs, such as specific S3 prefixes and KMS keys, to limit blast radius if credentials are compromised.

Model card

Documentation of a model's intended use, training data, performance metrics, limitations, and ethical considerations, delivered as part of the handoff.

Rollback runbook

A documented procedure listing triggers and steps to revert a deployed model to a prior approved version, including traffic shifting and verification checks.

Data drift

A change in the statistical distribution of input features relative to the training baseline. It can degrade predictions even when labels are unchanged.