Machine Learning and Data Processing
Core concepts
Machine learning fits models to data to make predictions or decisions. Know the three paradigms: supervised (labeled data; classification and regression), unsupervised (structure finding; clustering, dimensionality reduction), and reinforcement (agent learns from reward signals). Understand the train/validation/test split and cross-validation as guards against optimistic evaluation, and know overfitting (memorizes training data, poor generalization) versus underfitting (too simple).
Data processing
Candidates should know the pipeline: collection, cleaning (missing values, duplicates, outliers), feature engineering, encoding categorical variables, scaling/normalization, and splitting. Recognize class imbalance and its remedies (resampling, class weights, better metrics). Understand data leakage-information from the test set or target sneaking into training features-as a leading cause of models that look great and fail in production.
Metrics and traps
Accuracy is misleading on imbalanced data; know precision (of predicted positives, how many are right), recall (of actual positives, how many were found), F1 (their harmonic mean), and ROC-AUC as threshold-independent ranking quality. Traps include: confusing precision and recall; assuming correlation implies causation; treating a high test score as proof of deployment readiness; and forgetting that garbage in, garbage out applies to labels too.
Rule of thumb
Match the metric to the cost of errors. Missing a fraud case (false negative) usually costs more than a false alarm, so favor recall; flagging benign emails as spam is annoying, so favor precision.
Worked example
A model flags 100 transactions; 20 are truly fraudulent, and there are 30 frauds total. Precision = 20/100 = 0.20; recall = 20/30 ≈ 0.67. High recall, low precision: it catches most fraud but alarms a lot.