Reliable data, features, and useful visualizations
Define the unit of observation and the time at which a prediction will be made. A policy record, a claim record, and a transaction record cannot be joined indiscriminately. A one-to-many join can multiply exposure or loss amounts and produce an apparently larger dataset with incorrect totals. Reconcile record counts, key uniqueness, aggregate amounts, and unmatched records before and after each transformation. Preserve a reproducible path from source data to the modeling table, including the reasons for exclusions and any corrections.
Availability matters as much as correlation. A variable can be highly predictive because it contains information that would not exist at the intended prediction time. Claim settlement status, a later investigation result, or a target-derived summary calculated across the full dataset can leak outcomes into training or validation. Establish the information cutoff explicitly. Compute imputations, scaling, feature selection, and learned encodings using the training portion within the relevant split, then apply the fitted transformation to held-out observations. A clean train-test split is ineffective if preprocessing has already used the test labels.
Inspect missing values rather than treating them as a single nuisance. Missingness can reflect a collection process, a genuinely inapplicable field, or a change in system behavior. An imputation strategy should fit the variable and intended use, and a missingness indicator may be informative. Neither dropping every incomplete record nor replacing every missing value with zero is automatically justified. Also distinguish an impossible value from a rare but genuine event. Insurance losses can have long tails, so removing all large claims merely to improve fit can destroy the target of interest.
Categorical features require careful handling of levels, rare groups, and unseen categories. One-hot encoding is suitable for many unordered labels, while assigning arbitrary integers can impose a false order in some models. Target encoding needs leakage controls, such as appropriately constructed out-of-fold training values. Feature engineering should use domain knowledge without smuggling the answer into a predictor. Ratios need meaningful denominators and safeguards for zeros; aggregated histories must respect the relevant observation and prediction dates.
Choose visualizations to answer a question. Histograms and empirical distributions reveal shape; scatterplots can show relationships; time plots reveal shifts; grouped plots can expose differences that an aggregate obscures. Include units, denominators, uncertainty, and the volume of observations supporting a group. A dramatic average in a tiny cell may be unstable. Avoid letting a truncated axis, inconsistent binning, or exposure imbalance create a false story. Good data preparation ends with a dataset that is auditable and usable at deployment, not simply a table that makes the training score look better.