Data Handling, Cleaning and Visualization
What this domain tests
Data Handling, Cleaning and Visualization tests whether you can take raw data from common sources, repair quality problems, summarise variables, and choose a chart that answers a question. Missing values and chart selection are frequent exam targets.
Core rules and concepts
- Sources include CSV/Excel files, SQL databases, APIs, web pages and sensors. Tidy data has one row per observation and one column per variable.
- Detect missing values with isnull().sum(). Drop rows or columns only when missingness is small and random. Impute with mean, median or mode, use forward fill for time series, or use a model-based method.
- Clean duplicates, fix data types, standardise category labels, and flag outliers with the IQR rule: below Q1 - 1.5 IQR or above Q3 + 1.5 IQR.
- Summarise with describe(), value_counts() and groupby().
- Chart choice: histogram for distribution, boxplot for spread and outliers, bar chart for categorical counts, line chart for time trend, scatter for two numeric variables.
Worked example
Ages are [19, 20, 21, 22, 60]. The mean is 28.4, but the median is 21. The value 60 is an outlier: Q1 = 20, Q3 = 22, IQR = 2, upper fence = 25. Filling a missing age with the median preserves the typical age better than filling with the mean, because the mean has been pulled upward by 60.
Traps that cost marks
Dropping every row with a missing value can bias the dataset. Mean imputation is poor for skewed data. A pie chart with many categories is hard to read. Correlation does not prove causation. Do not remove an outlier automatically; first decide whether it is an error or a real extreme case.