H2O.ai Certified Machine Learning cheat sheet

Bias-variance decomposition: expected squared error = bias^2 + variance + irreducible noise; high bias underfits training and validation, high variance shows a large train-validation gap.

Confusion-matrix metrics: precision = TP/(TP+FP), recall = TP/(TP+FN), F1 = 2PR/(P+R); accuracy alone misleads when class prevalence is below roughly 10 percent.

ROC-AUC equals the probability a random positive outranks a random negative and is prevalence-insensitive; PR-AUC tracks minority-class performance and should be preferred for rare events.

H2O-3 binary classifiers emit a default 0.5 threshold; F1-optimal or cost-optimal thresholds are chosen from the threshold table, not assumed.

k-fold cross-validation reduces variance of the error estimate by a factor near 1/k; use stratified folds for classification and grouped folds when rows share an entity ID.

Regression metrics: MAE is in target units and robust to outliers, RMSE penalizes large residuals quadratically, R^2 = 1 - SS_res/SS_tot can go negative on validation data.

Log loss = -(1/n) * sum of [y*log(p) + (1-y)*log(1-p)]; it punishes confident wrong probabilities and is the default stopping metric for H2O-3 classification.

Distribution shift is diagnosed by comparing train and validation feature distributions and by a validation score that degrades while training score stays flat.

H2O-3 runs as a JVM cluster: a REST API layer plus distributed in-memory column-compressed frames; clients (Python, R, Java, Flow) are thin wrappers around REST calls.

H2O-3 is not a database; data must fit in aggregate cluster memory, and frames are moved to the cluster with import_file or upload_file rather than read lazily.

Driverless AI is a separate product from H2O-3: it automates feature engineering, model selection, and tuning, and produces a portable scoring pipeline artifact.

Driverless AI experiment artifacts include the MOJO pipeline and a Python scoring module; the MOJO is the deployment-ready form and needs no training environment.

H2O-3 MOJOs are generated by download_mojo or download_mojo_pojo and are scored with the h2o-genmodel runtime, independent of the training cluster.

H2O Wave is the application layer for building dashboards and apps over H2O models; it is not the model-serving runtime itself.

Cluster sizing rule of thumb: total memory must exceed the largest frame plus model working memory; XGBoost and GLM need less overhead than deep learning or stacked ensembles.

H2O-3 clusters are started via h2o.init(nthreads, max_mem_size); external backends (Hadoop, Sparkling Water) change the execution engine but keep the same REST and frame API.

H2O-3 infers column types on import; asfactor converts a numeric label to enum, and asnumeric converts a mis-typed numeric column before modeling.

Missing-value handling: numeric NAs are imputed (mean/median) by tree and GLM defaults, categorical NAs become a separate level; fillna and h2o.impute make the choice explicit.

Target leakage shows up as a feature with near-perfect univariate association to the label; drop it before training, since cross-validation will not reveal it.

High-cardinality categoricals (thousands of levels) inflate memory and overfit trees; options are target encoding, frequency encoding, or grouping rare levels.

Collinearity: variance inflation factor above about 10 signals redundancy; GLM with regularization or dropping one of a correlated pair resolves it.

Outlier treatment options are clipping at percentile bounds, log or Box-Cox transforms for skewed positives, and rank transforms for scale-free comparison.

Feature scaling matters for distance- and gradient-based models (GLM with regularization, k-means, deep learning) and is irrelevant for tree splits.

Driverless AI performs automatic feature evolution (transformations, interactions, target encoding) inside the experiment; the resulting engineered features are frozen into the MOJO.

H2O AutoML trains a fixed baseline set (GLM, default XGBoost, default GBM, default DRF) then random grids, then stacked ensembles; max_models or max_runtime_secs bounds the run.

H2O AutoML stacking uses cross-validated predictions as level-one features; the metalearner is typically a GLM, and the ensemble is only as good as its fold consistency.

Early stopping in H2O-3 GBM and XGBoost uses a validation frame and stopping_rounds, stopping_metric, and stopping_tolerance; without a validation frame it does not trigger.

Key GBM hyperparameters: ntrees, max_depth, min_rows, learn_rate, sample_rate, col_sample_rate; lower learn_rate requires more trees and typically generalizes better.

Grid search in H2O-3 is defined with H2OGridSearch over a hyperparameter map; random search is preferred when the space is large and evaluation is expensive.

XGBoost in H2O-3 exposes hist and approx tree methods; the hist method is the fast default for wide data, and exact is slow but precise on small data.

Regularized GLM tuning sweeps alpha (mixing of L1/L2) and lambda; lambda_search finds the path efficiently instead of a full grid.

Reproducibility requires setting a seed and a fixed fold assignment; AutoML fold assignment is stored in the leaderboard and must be reused for fair comparison.

Variable importance in H2O-3 trees is split-based (relative influence); it is biased toward high-cardinality features, so corroborate with permutation importance.

Permutation importance measures the drop in a chosen metric when a feature's values are shuffled; it is model-agnostic and reflects predictive reliance, not causality.

Partial dependence plots show the average model response as one feature varies; they assume feature independence and mislead under strong correlation.

SHAP values are additive per-row attributions summing to the prediction minus the baseline; TreeSHAP gives exact values for tree ensembles in polynomial time.

H2O-3 provides reason codes (per-row contributions) for GLM, GBM, DRF, XGBoost, and stacked ensembles; they are the per-prediction counterpart to global importance.

Individual conditional expectation curves disaggregate a partial dependence plot by row, exposing heterogeneous effects that the average hides.

Residual analysis: plot residuals against each feature and against predictions; structure in residuals indicates an unmodeled interaction or a missing transform.

Fairness and subgroup checks compare error rates across protected segments; a model can be accurate overall while failing one segment badly.

MOJO scoring is single-threaded per row but thread-safe; throughput scales with concurrent requests, and latency is microseconds to low milliseconds for tree models.

The h2o-genmodel Java runtime plus the MOJO zip is the minimal serving artifact; the training cluster can be shut down entirely after export.

POJOs are compiled Java classes tied to the H2O version; MOJOs are version-tolerant and are the recommended artifact for long-lived production.

Scoring parity check: score a holdout set in the training cluster and via the production path, then compare predictions within a tight tolerance before cutover.

Model drift monitoring compares live feature distributions and prediction distributions against training baselines (PSI or KS); retrain triggers are set on those thresholds.

A model registry stores the artifact, training data fingerprint, hyperparameters, metrics, and version; promotion between stages is an auditable state change.

Batch scoring via h2o.predict on a frame is the simplest path; real-time scoring needs a REST wrapper (for example a MOJO served behind an HTTP endpoint).

Rollback plan: keep the previous MOJO version deployable and route traffic by version tag; a failed parity or drift check reverts without retraining.