Designing an AI Solution Using Machine Learning
Domain 2: Designing an AI Solution Using Machine Learning
This domain tests solution design: framing the business problem, defining success, preparing data, choosing and validating a model, and documenting trade-offs. The exam expects you to follow a lifecycle such as CRISP-DM: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. You must justify decisions at each phase rather than jump to an algorithm.
Design rules
- Start with a measurable business objective and a baseline. Define the unit of prediction, label, time horizon, and the decision the model will support.
- Separate training, validation, and test data. For time-dependent problems, split by time, not randomly.
- Prevent leakage: no target-derived features, no future information, and no preprocessing fitted on the test set.
- Choose metrics from the decision: precision, recall, or F1 for imbalanced classification; MAE or RMSE for regression; lift or cost-based measures for business impact.
- Address class imbalance with resampling, class weights, or threshold tuning, not accuracy alone.
- Document assumptions, data lineage, feature definitions, and acceptance criteria.
Worked example
A telecom wants to reduce churn with a retention offer. The label is churned within 60 days, so this is binary classification. The business metric is retained revenue per contact, not raw accuracy. Because only 4% churn, accuracy would look high by predicting no churn for everyone; use precision at a fixed recall or expected value per offer. Split customers by observation date to avoid leakage. Build a logistic-regression baseline, then compare a gradient-boosted model using cross-validation. Set a threshold from campaign capacity and offer cost. Validate on a holdout period and report confusion-matrix costs.
Traps
Optimizing accuracy on imbalanced data; leaking future behavior into features; tuning on the test set; ignoring the baseline; treating correlation as causation; and failing to connect model metrics to the business decision. Also watch for labels that are proxies and may encode bias.