H2O.ai Certified Machine Learning common mistakes
H2O.ai Certified Machine Learning · Understand the traps and how to avoid them.
Treating the H2O-3 training frame and the leaderboard's cross-validation metrics as if they came from the same data partition, then reporting CV AUC as the model's generalization estimate after training on the full frame.
H2O's AutoML and Grid Search write cross-validation metrics into the leaderboard, but the final model returned by get_best_model() may be refit on 100% of the training frame. Candidates read the leaderboard AUC and assume it is a holdout score, because in scikit-learn the analogous cross_val_score result is a pure CV estimate.
What to do instead
Distinguish the two artifacts explicitly: use aml.leaderboard / model.cross_validation_metrics() for the CV estimate, and always score a held-out frame with model.model_performance(test_frame) before quoting a number. When comparing models, compare on the same test frame with the same metric and note whether the leader is a CV-refit or a single-split model.
Reading a confusion matrix at the default 0.5 threshold and reporting accuracy for an imbalanced binary problem, instead of reporting AUC, AUCPR, or a threshold chosen from the validation set.
H2O prints a threshold table but the headline confusion matrix defaults to max F1 or 0.5 depending on the call, and candidates assume the printed accuracy is the model's quality. With a 2% positive rate, a majority-class predictor looks 98% accurate and the candidate concludes the model is good.
What to do instead
For imbalanced targets, report AUC and area under the precision-recall curve from model_performance(), then pick an operating threshold from the validation set using the threshold table (perf.metrics() with thresholds_and_metric_scores()), and state the precision/recall trade-off at that threshold rather than at 0.5.
Comparing H2O-3 models across runs without controlling the fold assignment, seed, or `fold_assignment`, and attributing the resulting AUC difference to the algorithm.
H2O's default fold_assignment is AUTO, which picks Random or Stratified based on the response type, and each run can shuffle differently. Candidates rerun AutoML with a new seed, see a 0.01 AUC swing, and conclude XGBoost beat GBM, when the difference is fold noise.
What to do instead
Fix seed, fold_assignment, and nfolds (or supply an explicit fold_column) before comparing algorithms. For a defensible comparison, use the same folds for every model and check the standard deviation of the CV metric across folds, reporting the difference against that spread.
Feeding a target-encoded or otherwise leaky feature computed on the full dataset into H2O-3 training, then being surprised when CV AUC is near 1.0 and test AUC collapses.
Candidates compute encodings or aggregations in pandas before uploading to the cluster, so the encoding already saw every row including the validation folds. H2O's internal CV cannot undo leakage that happened upstream in the client-side preprocessing.
What to do instead
Compute any target-dependent transform inside the fold structure, or use H2O's built-in target encoding (available for GLM/XGBoost/DRF in recent H2O-3) which is fit per fold. For manual encodings, split first, fit the encoder on train only, and apply to validation/test; verify by checking that no single feature has suspiciously dominant importance.
Passing a column with high cardinality as a categorical to H2O-3 and expecting it to be handled like a numeric ID, or conversely leaving an ID column in the predictors.
H2O infers column types on import, and string columns become enums regardless of cardinality. Candidates either let a 50,000-level customer ID become a categorical predictor (exploding memory and overfitting) or leave a monotonic row number in, which the model happily splits on.
What to do instead
After h2o.import_file(), inspect frame.types and frame.cardinality(); drop or hash high-cardinality identifiers, and explicitly set column_types or convert with frame['col'].asfactor() only where the enum semantics are real. In Driverless AI, mark ID columns as ID/Text in the dataset settings so they are excluded from modeling.
Assuming H2O-3 AutoML tunes hyperparameters extensively by default, then reporting its leaderboard as a tuned baseline.
H2O-3 AutoML runs a fixed set of models with mostly default hyperparameters and only adds a light grid for stacked ensembles and a few algorithms; the heavy search is in Driverless AI. Candidates expect AutoML to do a full random search and are surprised that a manual grid beats it.
What to do instead
Know the distinction: H2O-3 AutoML is a model-and-ensemble selection procedure with limited hyperparameter variation, while Driverless AI performs evolutionary feature engineering and hyperparameter search. If tuning is required, run an explicit H2OGridSearch or RandomDiscretizedSearch after AutoML and compare on the same folds.
Interpreting H2O-3 variable importance from a tree model as a causal or directional statement, and ignoring that it is computed on training data with possible correlated-feature splitting.
H2O reports split-based importance (or permutation importance if requested) that sums gains across correlated features arbitrarily; candidates read 'feature X is most important' as 'X drives the outcome' and omit the direction or the correlation caveat.
What to do instead
Pair importance with a directional method: use partial dependence (model.partial_plot()) or SHAP (model.shap_summary_plot() / predict_contributions) to show the sign and shape of the effect, and check for correlated predictors before attributing importance to a single variable. State that importance is associational.
Explaining a stacked ensemble or a Driverless AI model by pointing at the base learners' importances without explaining the meta-learner, or treating Driverless AI's MLI as the model's own reasoning.
Stacked ensembles in H2O-3 combine base model outputs through a metalearner, so base-model importances do not describe the ensemble's behavior. Driverless AI's MLI (K-LIME, Shapley, decision-tree surrogates) are post-hoc approximations, and candidates present them as exact.
What to do instead
For H2O-3 stacks, explain at the level of the metalearner inputs and use predict_contributions on the ensemble where supported; for Driverless AI, label MLI outputs as surrogate/approximate explanations, report the surrogate's fidelity, and prefer reason codes (K-LIME) for per-row explanations when the use case is adverse-action style.
Deploying a model by saving the H2O-3 binary and loading it in a fresh process without matching the H2O version, or assuming a POJO/MOJO will reproduce predictions without the same preprocessing.
H2O-3 model binaries are version-sensitive and require a running H2O cluster; POJOs/MOJOs are version-tagged and the scoring pipeline must replicate exactly the same imputation, encoding, and column order used at training. Candidates hit 'model trained with version X, current version Y' or silently different predictions.
What to do instead
For production, export a MOJO (h2o.download_mojo()) and score with the MOJO runtime, pin the H2O version in the runtime image, and persist the exact preprocessing (or use the MOJO's built-in preprocessing where available). Validate by scoring the same rows in the training cluster and in the deployment runtime and comparing predictions to a tight tolerance.
Treating a Driverless AI experiment's validation score as a production estimate without accounting for the time-series split, the holdout, or the reproducibility settings used, and skipping the scoring pipeline artifact.
Driverless AI experiments can use time-based validation, k-fold, or a final holdout, and the reported score depends on which. Candidates quote the experiment's validation metric and deploy the model without exporting the Python scoring pipeline or the MOJO, so the production transform diverges from the experiment's feature engineering.
What to do instead
Read the experiment's validation scheme and report the metric with its scheme named; use the experiment's holdout or an external test set for the production estimate. Export the scoring pipeline (Download Scoring Pipeline) or MOJO together with the model, and run the exported artifact on a sample to confirm it reproduces the experiment's predictions before promoting.