Intel MLOps Professional cheat sheet

MLOps maturity levels (Google-style): Level 0 manual, script-driven, no CI/CD; Level 1 automated ML pipeline with continuous training; Level 2 full CI/CD for pipelines plus automated trigger and rollback.

CRISP-DM phases: business understanding, data understanding, data preparation, modeling, evaluation, deployment; iteration is expected between preparation and modeling, not a single pass.

Scoping decision rule: if the label can be computed deterministically from logged data, prefer a supervised batch pipeline; if labels arrive delayed, use proxy labels plus a scheduled relabeling job.

Build-vs-buy test: reuse an existing governed feature store or registry when the new workload shares >60% of its entities and freshness SLAs; otherwise a dedicated pipeline is cheaper to operate.

Latency scoping: real-time serving (single-request, p99 in milliseconds) vs batch scoring (throughput-bound) vs streaming (windowed micro-batch); the choice constrains the whole stack, so fix it before model selection.

Business metric to model metric mapping must be explicit: e.g. fraud detection uses recall at a fixed false-positive budget, not raw accuracy, because class imbalance makes accuracy misleading.

Total cost of ownership levers for ML: GPU/CPU utilization rate, feature recomputation cost, storage of experiment artifacts, and inference instance count; idle accelerators are the largest avoidable cost.

Risk register items to scope early: data drift owner, model rollback SLA, human-in-the-loop requirement, and regulatory classification (e.g. high-risk AI under the EU AI Act) which mandates documentation and logging.

Point-in-time correctness: training features must be computed as of the prediction timestamp, not the label timestamp; joining a label later than the feature snapshot prevents leakage.

Feature store contract: offline store for training (columnar, partitioned by event date) and online store for serving (low-latency KV); both must be populated from the same transformation code to avoid train/serve skew.

Temporal splits are mandatory for time-dependent data: train on past, validate on future; random splits leak future information when rows are autocorrelated.

Data quality gates: schema conformance, null-rate thresholds, range checks, cardinality checks, and freshness (max lag) - each gate should fail the pipeline loudly rather than silently drop rows.

Parquet with partition pruning on event date and column pruning reduces scan volume; small-file problems are fixed by compaction/OPTIMIZE jobs rather than by more partitions.

Idempotent ingestion: writes keyed by (entity_id, event_ts) with upsert/merge semantics allow safe replays; append-only without dedup produces duplicated training rows.

Feature versioning: a feature definition change (e.g. unit change, window change) requires a new version name; overwriting in place silently invalidates historical training sets.

Leakage detectors: single-feature near-perfect AUC, features whose availability timestamp is after the label, and features computed over the full dataset rather than per-window.

Reproducibility triad: code commit hash, data snapshot/version, and full hyperparameter + environment spec; missing any one makes a result non-reproducible.

Experiment tracking record should include run id, params, metrics per epoch, artifact URIs, git SHA, random seeds, and hardware (CPU/GPU, driver versions).

Baseline-first rule: a trivial baseline (majority class, last value, linear model) must be logged before complex models; gains are reported relative to it.

Cross-validation choice: k-fold for i.i.d. data, grouped k-fold when entities repeat, time-series split when ordering matters; choose based on data dependence structure, not convenience.

Metric selection: use PR-AUC for rare positives, ROC-AUC when classes are balanced, MAE/RMSE for regression, and report calibration (reliability curve, Brier score) when probabilities drive decisions.

Hyperparameter search budget: random search outperforms grid search when few hyperparameters matter; successive halving or Bayesian optimization when each trial is expensive.

Model registry states: staging → production → archived, with immutable versions and lineage links to the training run and data snapshot; promotion requires passing evaluation gates.

Seed control: set Python, NumPy, framework, and data-loader seeds; note that GPU nondeterminism (atomic ops) may still cause small run-to-run variance.

Intel oneAPI Deep Neural Network Library (oneDNN) is the CPU backend used by TensorFlow and PyTorch on Intel CPUs; oneDNN Graph fuses operator patterns to cut memory traffic.

Intel Extension for PyTorch (IPEX) provides CPU optimizations and bfloat16/INT8 paths; typical usage is importing the extension before model creation and using its optimizer wrapper.

Intel Neural Compressor (INC) performs post-training quantization, static/dynamic quantization, and quantization-aware training, plus accuracy-aware tuning with a configurable tolerance.

Quantization decision rule: INT8 on CPU typically gives 2-4x speedup with small accuracy loss; validate on a representative calibration set and set an accuracy-drop threshold before deployment.

bfloat16 keeps the FP32 exponent range, so it usually needs no loss scaling; FP16 is narrower in range and typically requires loss scaling to avoid underflow.

OpenVINO converts models to an Intermediate Representation and adds runtime optimizations (layer fusion, layout optimization); its benchmark_app reports latency and throughput per inference request.

Intel Distribution of OpenVINO toolkit Model Optimizer accepts ONNX, PaddlePaddle, TensorFlow, and PyTorch-exported graphs; dynamic shapes should be declared at conversion to avoid recompilation.

Optimization order of impact: better data/model architecture, then mixed precision, then quantization, then graph/runtime tuning; optimizing an over-parameterized model first wastes effort.

Roofline reasoning: if arithmetic intensity is low the workload is memory-bound, so reducing precision (bytes moved) helps more than adding compute.

Serving patterns: real-time REST/gRPC, batch scoring, and streaming; each needs different autoscaling signals (request concurrency, queue depth, or window backlog).

KServe/InferenceService provides a standard predictor/transformer/explainer split with scale-to-zero; transformer containers run pre/post-processing next to the model.

Canary and shadow deployment: shadow mirrors live traffic without serving results, canary routes a traffic percentage; promotion criteria are error rate, latency p99, and business metric parity.

Container best practice: pin base image digests, install only runtime dependencies, run as non-root, and bake the model artifact or pull it from a versioned registry path.

Orchestration DAG rules: tasks must be idempotent and retry-safe; use sensors for data arrival, and set explicit timeouts and SLA alerts rather than relying on default retries.

Resource sizing: set CPU/memory requests from measured p95 usage and limits with headroom; missing requests cause noisy-neighbor eviction, oversized requests waste cluster capacity.

Rollback design: keep the previous model version warm and switchable by config or traffic split so recovery does not require a full redeploy.

Health endpoints: separate liveness (process up) from readiness (model loaded, dependencies reachable) so orchestrators do not route traffic to a warming replica.

Data drift vs concept drift: data drift is a change in input distribution P(X); concept drift is a change in P(Y|X), which input monitoring alone cannot detect.

Population Stability Index: PSI = Σ (actual% − expected%) × ln(actual%/expected%); common thresholds are <0.1 stable, 0.1-0.25 moderate, >0.25 significant shift.

Kolmogorov-Smirnov test compares continuous distributions; chi-square compares categorical frequencies; Jensen-Shannon divergence is symmetric and bounded, useful for binned features.

Performance monitoring needs labels; when labels are delayed, monitor proxies (prediction distribution, confidence scores, input drift) and schedule delayed-label backfill evaluation.

Four golden signals for model services: latency, traffic, errors, saturation; add model-specific metrics: prediction rate by class, null/unknown rate, and feature-missing rate.

Alert design: alert on sustained threshold breaches over a window, not single points; pair every drift alert with an owner and a documented remediation playbook.

Logging for observability: capture request id, model version, input feature vector or hash, prediction, and latency; retain enough to reconstruct decisions for audit.

Retraining triggers: scheduled cadence, drift threshold breach, performance degradation beyond tolerance, or upstream data schema change.

CI for ML adds: data validation tests, model quality tests, and artifact build; CD adds: model registration, staged rollout, and automated rollback on metric regression.

Pipeline triggers: code commit (CI), new data arrival (continuous training), drift alert, and scheduled retraining; each trigger needs its own guardrails and approval policy.

Testing pyramid for ML: unit tests for transforms, schema/contract tests for data, integration tests for pipeline, and model behavior tests (invariance, directional expectation, minimum performance).

Governance artifacts: model card (intended use, training data, metrics, limitations), data sheet, approval record, and lineage from data snapshot to deployed version.

Approval gates: automated quality gate (metric threshold, bias check, latency budget) followed by human sign-off for high-risk models; gate results must be stored with the release.

Environment promotion: dev → staging → prod with identical container images; only configuration and model version differ, which removes 'works in staging' drift.

Audit trail requirements: who approved which version, when, against which evaluation report; immutable logs and signed artifacts support this.

Automation anti-pattern: retraining on a schedule with no quality gate can silently deploy a worse model; always compare candidate against the current production baseline.

Threat surface for ML: data poisoning, model evasion, membership inference, model extraction, and artifact/supply-chain tampering; each maps to a distinct control.

Membership inference mitigation: limit per-sample influence (regularization, differential privacy), avoid overfitting, and restrict prediction confidence output when not needed.

Differential privacy: epsilon bounds privacy loss; smaller epsilon means stronger privacy and more utility loss; the privacy budget composes across queries, so track cumulative epsilon.

PII handling: tokenize or hash identifiers before feature computation, keep a separate restricted mapping table, and enforce purpose limitation on downstream joins.

Bias evaluation: compare subgroup metrics (demographic parity difference, equalized odds difference) against thresholds; a model can be accurate overall and unfair per group.

Explainability: SHAP or permutation importance for tabular models, saliency or attention for text/image; explanations should be validated for stability, not just generated.

Supply chain: pin and scan dependencies (SBOM), verify model artifact signatures, and restrict registry write access; an unsigned model is an unverified executable.

Access control: least privilege for feature stores and registries, separate service accounts per pipeline, and secrets from a vault rather than baked into images or notebooks.