Predictive AI
Predictive AI
The Predictive AI domain tests the AutoML workflow end to end: framing the problem, preparing data, choosing validation, comparing models, and deploying. On DataRobot, expect terminology such as project target, Autopilot, Leaderboard, Blueprint, Feature Discovery, and Deployment. The exam favors judgment about leakage and metric fit over memorized button locations.
Core rules
- Define the unit of prediction, the target, and the decision the model will support.
- Choose a split that matches deployment. Time-dependent problems need time-aware or out-of-time validation, not a random split.
- Use Autopilot to establish a baseline, then compare blueprints on a metric tied to the business decision, not just the Leaderboard default.
- Audit features for leakage: fields populated after the outcome, IDs, and proxies for the target.
- Check class imbalance, threshold selection, and subgroup performance before deployment.
- Confirm that deployment-time features are available and computed the same way as training features.
Worked example
A churn project uses 18 months of customer data. A random split gives AUC 0.91; an out-of-time split gives 0.78. The top feature is cancellation_reason, which is recorded only after a customer cancels. Removing it drops the random-split AUC to 0.80 and the out-of-time AUC to 0.77. The team selects a model at 0.77 and sets the outreach threshold from campaign capacity, not from the default 0.5. Reasoning: the first result was leakage, not skill. The out-of-time result is the honest estimate, and threshold choice is a business constraint.
Traps
- Random-splitting time series or event data and reporting an inflated score.
- Optimizing AUC when the business cares about precision at a fixed review capacity.
- Keeping leakage features because "the model found them useful."
- Ignoring missing-at-deployment features; a blueprint can score well and fail in production.
- Treating the Leaderboard winner as the deployment winner without subgroup and stability checks.