I85PREP
⌘ K
H2O.ai Certified Machine Learning study materials
H2O.ai Certified Machine Learning study resources

H2O.ai Certified Machine Learning: Model Training, Tuning, and AutoML

H2O.ai Certified Machine Learning · Work through your exam, one topic at a time.

All guide sections

Model Training, Tuning, and AutoML

What this domain tests

Model Training, Tuning, and AutoML tests whether you can configure H2O-3 algorithms, run a defensible search, and use AutoML without mistaking the leaderboard for a production decision.

Core rules

  • H2O-3 provides GLM, GBM, XGBoost, DRF, Deep Learning, Stacked Ensembles, and other algorithms. Key controls include ntrees, max_depth, learning_rate, sample_rate, col_sample_rate, stopping_rounds, stopping_metric, and score_tree_interval.
  • Use a validation frame or cross-validation for early stopping; do not stop on the test set. Set a seed and record it for reproducibility.
  • h2o.grid() supports explicit grids and RandomDiscrete search. Use search_criteria with max_models or max_runtime_secs, and keep the search space documented.
  • h2o.automl() trains and ranks models, including stacked ensembles. Important arguments include max_models, max_runtime_secs, nfolds, stopping_metric, sort_metric, exclude_algos, and seed.
  • Driverless AI experiments expose accuracy, time, and interpretability settings; feature engineering and model recipes are part of the experiment. Reproducibility depends on recording settings, data version, and recipe versions.
  • A stacked ensemble should be built from out-of-fold predictions; reuse the same folds when comparing base models.

Worked example

Run h2o.automl() with max_models = 20, max_runtime_secs = 600, nfolds = 5, sort_metric = 'AUC', and seed = 1. The leaderboard ranks by AUC, but the business cares about recall at a fixed review budget. Take the top three models, evaluate precision/recall on the untouched test set at the operating threshold, and choose the one with the lowest cost. The reasoning is that the leaderboard's sort metric is not automatically the business metric.

Traps

  • Assuming max_runtime_secs guarantees a fixed number of models.
  • Selecting the leaderboard winner without a holdout check.
  • Using the test set for early stopping or grid selection.
  • Changing folds between base models and a stacked ensemble.
  • Forgetting that AutoML and Driverless AI have different search spaces and artifacts.