ICML common mistakes
ICML · Understand the traps and how to avoid them.
Computing a classification metric such as F1 or accuracy on label-encoded multiclass predictions without checking whether the metric is macro-, micro-, or weighted-averaged, then reporting a single number as if the averaging choice were irrelevant.
Candidates memorize the F1 formula for binary problems and mechanically extend it to multiclass data, not realizing that macro-averaging weights rare classes equally while micro-averaging is dominated by frequent classes, so the reported score can shift dramatically with the same predictions.
What to do instead
Before computing any multiclass metric, explicitly state the averaging strategy and justify it from the class balance and the cost of errors; when in doubt, report macro and micro side by side and explain the discrepancy.
Applying a feature-scaling or imputation step to the full dataset before the train/test split, then reporting cross-validation scores that are inflated relative to true generalization.
Pipelines are often written as a linear script where preprocessing feels like 'cleaning' that logically precedes modeling, and the leakage is invisible because the code runs without error and produces plausible-looking scores.
What to do instead
Wrap all learned preprocessing (scalers, imputers, encoders, target encoders) inside a Pipeline or ColumnTransformer and fit it only on training folds; audit by checking that no transformer's fit sees test rows.
Interpreting a model's coefficients or feature importances as causal effects or as evidence that a feature is 'important to the outcome' without considering correlated predictors.
Coefficients are the most convenient numeric output of a linear model, and the intuition that a large weight means a large effect is reinforced by textbook examples that use uncorrelated features.
What to do instead
Check for multicollinearity (VIF, correlation matrix) and use permutation importance or conditional importance for a predictive read; state explicitly that these are associational and require a causal design to support causal claims.
Choosing the number of clusters by eyeballing a scatter plot or accepting the default k, without using an internal criterion or domain justification, and then treating the resulting cluster labels as ground-truth classes.
Clustering has no labels to validate against, so candidates default to visual inspection on two dimensions, which does not generalize to high-dimensional data and hides instability.
What to do instead
Compare k across a range using silhouette, Davies-Bouldin, or gap statistics, check stability across seeds and bootstrap samples, and present clusters as exploratory structure with a stated validation rationale.
Reporting a single train/validation split score as the expected test performance, especially on small datasets, without any estimate of variance.
A single number is easy to produce and easy to read, and the variance of a holdout estimate on a few hundred examples is large but not visible in the output.
What to do instead
Use k-fold cross-validation with repeated splits where feasible, report mean and standard deviation across folds, and comment on whether the spread is small enough to support the claimed comparison.
Diagnosing overfitting or underfitting from training loss curves alone while ignoring the validation curve's shape, learning-rate schedule, and batch-size effects.
Training loss is the first quantity printed and always decreases, so candidates anchor on it and interpret any gap as overfitting without considering that a still-decreasing validation loss indicates underfitting.
What to do instead
Plot train and validation curves together, identify the regime (both high, gap widening, both low) before prescribing a remedy, and pair the diagnosis with a concrete intervention such as regularization, more capacity, or early stopping.
Describing a deployment plan that ends at 'train the model and serve it' with no monitoring, retraining trigger, or rollback path, or conversely proposing a complex MLOps stack with no justification tied to the problem.
Coursework rewards model accuracy and rarely exercises the operational lifecycle, so candidates treat deployment as an afterthought and reach for buzzwords when prompted.
What to do instead
Structure the answer around data drift detection, performance monitoring with alert thresholds, a retraining cadence tied to drift or label latency, versioned artifacts, and a rollback procedure, each justified by the specific failure it prevents.
Treating fairness as satisfied by removing a protected attribute from the feature set, without examining proxy variables or measuring disparate impact across groups.
Fairness-by-exclusion is the most commonly taught shortcut and is easy to implement, and its failure is invisible unless subgroup metrics are explicitly computed.
What to do instead
Measure performance and selection rates by group, test for proxy leakage from correlated features, and name the specific fairness criterion (e.g., equalized odds, demographic parity) and its trade-off with accuracy.
Handling missing data by dropping all rows with any missing value or filling with zero, without checking the missingness mechanism or the resulting distribution shift.
Listwise deletion and zero-fill are one-line operations that produce a clean-looking dataframe, and the bias they introduce is not visible in the output.
What to do instead
Characterize missingness (MCAR/MAR/MNAR) and the fraction per column, prefer model-based or iterative imputation with an added missingness indicator, and compare the imputed distribution against the original.
Answering a probability or estimation question with a point estimate only, ignoring the question's request for uncertainty, or confusing a confidence interval's interpretation with a credible interval's.
Point estimates are the default output of most modeling code, and the frequentist/Bayesian distinction is often memorized as vocabulary rather than as a difference in what the interval means.
What to do instead
Read the prompt for the estimand and the uncertainty requirement, then state the interval with its correct interpretation: a confidence interval concerns repeated-sampling coverage of the procedure, a credible interval is a posterior statement about the parameter.