ML Development Testing
Two intertwined test targets
ML development testing covers (a) the conventional code - data pipelines, feature computation, training scripts, serving APIs - and (b) model behaviour. Unit and integration testing apply to (a) in the usual way; (b) needs metric-based and statistical approaches.
What to test
- Pipeline code: schema validation, null handling, feature transformations, joins, aggregation logic.
- Feature consistency: training/serving skew - the feature computed offline must match the one computed online at inference time.
- Reproducibility: version data, code, hyperparameters, library versions and random seeds. Without them, a defect cannot be re-created.
- Non-determinism: identical inputs may give different outputs across runs or after retraining; assertions need tolerances or statistical bounds, not exact expected values.
- Model comparison: use resampling, confidence intervals or significance tests before declaring improvement.
- Deployment: shadow deployment (run silently alongside the incumbent), canary release, A/B testing, then continuous monitoring.
The oracle problem and traps
Traditional expected-value assertions break down for learned components. Prefer property-based and metamorphic assertions ("output is a probability in [0,1]", "sum of class probabilities is 1", "a translated image yields the same class").
Frequent exam traps:
- Expecting bit-identical model output across runs.
- Testing only the notebook/model and not the data pipeline or serving path.
- Declaring model B better than A from one training run.
- Forgetting that CI/CD for ML must also re-run data validation and model evaluation gates.
Worked example
A feature avg_spend_30d is computed by SQL offline and by Java streaming online. Both are "unit tested" in isolation and pass, but they define the 30-day window differently (inclusive vs exclusive end date). Only a consistency test comparing offline and online values for the same entity detects the skew.