I85PREP
⌘ K
H2O.ai Certified Machine Learning study materials
H2O.ai Certified Machine Learning study resources

H2O.ai Certified Machine Learning: Machine Learning Fundamentals and Model Evaluation

H2O.ai Certified Machine Learning · Work through your exam, one topic at a time.

All guide sections

Machine Learning Fundamentals and Model Evaluation

What this domain tests

The Machine Learning Fundamentals and Model Evaluation domain checks whether you can frame a supervised problem, choose a validation design, and pick the metric that matches the business decision. Questions often contrast accuracy with precision/recall, ask why k-fold cross-validation is used, and probe leakage.

Core rules

  • State the unit of prediction, the target, and the cost of each error type before choosing a model.
  • Split data into train, validation, and test. Use k-fold cross-validation for model selection; keep the test set untouched until the final check.
  • For classification, stratify folds when classes are rare. In H2O-3, set nfolds, fold_assignment = 'Stratified', and keep_cross_validation_predictions = TRUE when you need out-of-fold predictions for stacking.
  • Accuracy is misleading under imbalance. Precision answers: of predicted positives, how many are right? Recall answers: of actual positives, how many were found? F1 balances them; PR-AUC is often more informative than ROC-AUC when positives are rare.
  • For regression, RMSE penalizes large errors more than MAE; R2 is relative to the mean baseline and can be negative on unseen data.
  • Compare models on the same split, same metric, and same population. A leaderboard is only a ranking under its sort_metric.

Worked example

A fraud model is tested on 10,000 transactions with 100 frauds. Model A predicts all negative: accuracy 99%, recall 0. Model B gives TP=50, FP=40, FN=50, TN=9,860. Accuracy is 99.1%, precision 50/90 = 55.6%, recall 50/100 = 50%, F1 = 52.6%. If a missed fraud costs 200 and a review costs 10, Model A costs 100 x 200 = 20,000; Model B costs 50 x 200 + 40 x 10 = 10,400. Model B is better despite almost identical accuracy. The reasoning is cost-weighted, not accuracy-weighted.

Traps

  • Reporting accuracy on a 1% positive rate and declaring victory.
  • Preprocessing before splitting, which leaks test information into training.
  • Tuning on the test set, then reporting that score as production performance.
  • Comparing ROC-AUC across datasets with different case mixes.
  • Using R2 for a nonlinear model as if it proved calibration.