Data Handling
Domain 4 - Data Handling. This domain tests whether you can take a messy dataset and produce a clean, correctly split table suitable for a model, and justify every cleaning decision you make.
Core rules and concepts
- Data types: structured (rows and columns: CSV, spreadsheets, SQL tables), semi-structured (JSON, XML), unstructured (images, audio, free text). Features are the inputs; the label or target is what is predicted.
- Quality checks: missing values, duplicate rows, inconsistent formats (Kolkata vs kolkata), out-of-range values, outliers, wrong dtypes such as a number stored as text.
- Missing data: drop rows only when few and missing at random; drop a column when missingness is very high; impute with the mean for symmetric data, the median for skewed data or outliers, the mode for categorical data, or add a binary missing-value flag.
- Encoding: one-hot encoding for nominal categories (no order); ordinal or label encoding only for genuinely ordered categories, otherwise the model invents a ranking.
- Scaling: min-max maps to [0, 1]; standardisation gives mean 0 and standard deviation 1. Fit the scaler on training data, then transform validation and test data.
- Splitting: train / validation / test, commonly 60-80 / 10-20 / 10-20, stratified when classes are imbalanced. The test set is opened once, at the end.
Worked example
A file of 1,000 student records has Age missing in 50 rows (5%) and annual family income missing in 400 rows (40%), with income heavily right-skewed. Decision: split first, then compute statistics on the training fold only. Impute Age with the median, which is robust to skew and only 5% is missing. For income, either drop the column or impute the median plus a binary income_missing flag, because filling 40% of a column invents more data than it recovers.
Traps that cost marks
- Imputing or scaling before the train-test split, which leaks test information into training.
- Mean imputation on skewed income; the mean sits above most of the data.
- Using roll number or student ID as a numeric feature.
- Dropping rows and accidentally deleting most of the minority class.