Hyperparameter Tuning Under OOM and Holdout Reuse
An ML team tunes an XGBoost classifier for loan default on 2 million rows and 120 features, including 18 categorical fields; one field has 50,000 distinct merchant codes. They launch 400 Hyperopt trials with SparkTrials on an 8-worker cluster, parallelism set to 8. About 35% of trials fail with out-of-memory errors. The best cross-validation log loss is 0.221, but the untouched holdout is 0.249. To choose the final model, the team repeatedly evaluates 30 candidate trials on the same holdout and picks the lowest score. Training currently converts the Spark DataFrame to a single pandas DataFrame on the driver, then one-hot encodes all categoricals. MLflow logs only the best trial, and the registered model has no signature or input example. The cluster has eight workers with 32 GB each and no GPU.
1. Should this workload use Hyperopt Trials or SparkTrials? Describe the configuration and tracking changes you would make.
Read the worked answer
Use SparkTrials when each trial can run on the cluster and you want parallel trials across workers. Hyperopt Trials runs on the driver and is appropriate for small data or single-node training, not 2 million rows with many trials. Configure SparkTrials with max_evals=400 and a parallelism that each worker can actually support; eight concurrent trials on eight workers may be too aggressive if each trial needs substantial memory. Reduce parallelism before adding workers. Use MLflow nested runs: one parent run for the search and one child run per trial, logging parameters, metrics, and the model artifact. Do not log only the best trial, because you lose the evidence needed to diagnose failures. Use early stopping on an inner validation fold so trials do not train to completion unnecessarily. After selecting parameters, retrain on the full training set and log a model signature and input example. Do not nest parallel Hyperopt searches inside SparkTrials.
2. Address the out-of-memory failures and the 50,000-category merchant field. What concrete training changes would you make?
Read the worked answer
The driver pandas conversion and one-hot encoding are likely the main causes. Collecting 2 million rows to the driver defeats distributed training and can exhaust driver memory. Use distributed XGBoost on Spark or keep data partitioned and let the learner handle it. For the merchant field, one-hot encoding to 50,000 columns is sparse and memory-heavy; replace it with frequency encoding, hashed features, target encoding fit inside cross-validation folds, or native categorical handling where supported. Set tree_method=hist and lower max_bin to reduce memory. Reduce max_depth and lower trial parallelism. Increase executor memory only after reducing per-trial footprint. Cache the prepared training matrix only if it fits; otherwise avoid materializing it. Monitor executor spill and failed-task logs. GPU is not automatically the answer: fix the data representation and concurrency first, then consider GPU if profiling shows the model fit is the bottleneck.
3. The team selects the best of 30 trials on the same holdout. Explain the statistical problem and propose a validation and registration protocol.
Read the worked answer
Repeatedly scoring 30 candidates on one holdout turns that holdout into a validation set. The selected score is optimistically biased, and the reported 0.249 is no longer an unbiased estimate. Use nested temporal cross-validation: outer folds estimate generalization, inner folds select hyperparameters and early-stopping rounds. Report mean and variance across outer folds, not just the best number. Choose the final configuration before touching the final holdout, then evaluate once. If the holdout result is materially worse than cross-validation, treat it as a distribution-shift warning, not a reason to tune again on the holdout. For registration, log the training data version, feature order, git commit, evaluation protocol, and signature inferred from the training DataFrame. Add an input example so serving validation catches schema mismatches. Register an immutable version and promote via alias after an integration test.