Input Data Testing
Why data is a test object
In ML the training data effectively is part of the specification, so data defects become product defects. Test data quality across dimensions such as completeness, accuracy, consistency, validity, uniqueness, timeliness and balance. Also test the labels: label noise, annotator disagreement, inconsistent labelling guidelines.
Representativeness and bias
A model cannot be better than the population it saw. Check that training, validation and test data come from the same distribution as the intended operational population; check subgroup coverage for protected or business-critical segments; check class imbalance. Sources of bias include historical bias in the data, sampling bias (convenience samples, sensor placement), labelling bias, and proxy features correlated with protected attributes.
Data drift and concept drift
- Data (covariate) drift: the input distribution changes; the input→output relationship is stable.
- Concept drift: the relationship itself changes (e.g. user behaviour after a policy change). Both invalidate evaluation results and require monitoring against a reference distribution.
Techniques
Profiling and statistical summaries; constraint/invariant checks (ranges, formats, referential integrity); schema and pipeline tests; duplicate and near-duplicate detection; distribution comparison between training and production data; per-slice performance monitoring as a data-quality signal.
Trap in questions
- Treating data quality as a one-off check before training rather than a continuous activity.
- Confusing data drift with concept drift (a favourite discrimination item).
- Assuming more data always helps - more biased or mislabeled data makes things worse.
- Ignoring the test set's own quality: an unrepresentative or leaked test set produces confident nonsense.