Model Development: Features, Training, and Tuning
Feature engineering
Spark ML models consume a single vector column, so numeric features are combined with VectorAssembler. Categorical strings go through StringIndexer then OneHotEncoder; scale with StandardScaler or MinMaxScaler when the algorithm is distance- or coefficient-sensitive. Wrap every step in a Pipeline so identical transformations apply at training and scoring time.
Splitting and evaluation
Split into train/validation/test (or use CrossValidator/TrainValidationSplit with an Evaluator such as BinaryClassificationEvaluator for ROC AUC or MulticlassClassificationEvaluator for F1). Fit transformers and the model on the training data only; fitting an indexer or scaler on the full dataset leaks test information.
Tuning
Hyperopt's fmin searches a space built from hp.uniform, hp.choice, hp.loguniform. Use SparkTrials to distribute trials across a cluster; use plain Trials on a single node. Set max_evals deliberately and record results in MLflow.
Traps
VectorAssembler rejects string columns, so encode first. StringIndexer fitted on train can throw on unseen labels during scoring unless handleInvalid is set. A pipeline must be fitted, not just assembled. Class imbalance calls for resampling or class weights, not a metric switch alone.
Rule of thumb
If a transformation changes the data, it belongs inside the pipeline; if it changes the search, it belongs inside the objective function passed to Hyperopt.