DataRobot Certified Professional common mistakes
DataRobot Certified Professional · Understand the traps and how to avoid them.
Treating agentic workflows like a single prompt chain: the candidate describes an agent as one LLM call that 'decides what to do next', without naming a planner/executor split, a tool registry, or a termination condition, so multi-step tasks with retries or branching are unanswerable.
Most candidates have only built RAG chatbots or single-turn assistants. The exam's agentic items ask about orchestration state, loop limits, and tool-selection failure, and a prompt-chain mental model has no vocabulary for those, so answers default to 'improve the prompt'.
What to do instead
Before the exam, sketch the lifecycle of one agent task end to end: goal intake, plan decomposition, tool selection with schema and permission checks, execution, observation of results, reflection or replanning, and a stop condition (step budget, confidence threshold, human handoff). Practise explaining where a loop can run away and what bounds it. When a question mentions tool errors or repeated actions, look for the answer about bounded iteration and fallback, not prompt wording.
Confusing guardrails with evaluations in generative AI answers: the candidate picks 'add a guardrail' when the scenario is about measuring output quality, or picks 'run an evaluation' when the scenario is about blocking a harmful response at inference time.
Both live near the model and both are described with words like 'safety' and 'quality'. Under time pressure the two collapse into one idea, and the candidate chooses whichever term appears in the stem.
What to do instead
Hold a clean split: guardrails act in the request/response path and enforce policy (blocking, redaction, topic restriction, PII filtering); evaluations run offline or in batch against a dataset and produce scores for quality, groundedness, and safety. Ask 'is this decision made per request in production, or across many examples before release?' That single question resolves most of these items.
Answering hallucination questions with 'lower the temperature' or 'use a bigger model' instead of grounding: the candidate never mentions retrieval, citation requirements, or refusing when context is insufficient.
Temperature and model size are the most familiar knobs, so they surface first. But the exam frames hallucination as a grounding and evaluation problem, and generic tuning answers are the distractor, not the key.
What to do instead
For any hallucination scenario, work through the grounding chain: retrieve authoritative context, constrain the prompt to that context, require citations to retrieved passages, instruct the model to decline when support is missing, then measure groundedness on a labeled set and monitor it in production. Choose the option that adds evidence and verification over the one that adjusts sampling parameters.
Treating feature drift and concept drift as the same thing: the candidate says 'retrain' for every drift question without distinguishing a shift in input distributions from a shift in the relationship between inputs and target.
Both are called drift and both eventually justify retraining, so the distinction feels academic. The exam, however, tests whether the candidate can pick the right diagnostic and the right response, and 'retrain' is often the wrong immediate action.
What to do instead
Separate the two explicitly. Feature drift means input distributions moved while the input-to-target relationship held; the fix is usually to investigate upstream data, check pipeline changes, and monitor performance before retraining. Concept drift means the relationship itself changed; performance degrades even on stable inputs and retraining on recent data is the real remedy. Read the scenario for whether model accuracy has actually fallen, and let that decide.
Choosing a model by accuracy alone on an imbalanced problem: the candidate selects the highest-accuracy model when the positive class is rare, ignoring precision, recall, and the business cost of each error type.
Accuracy is the default metric taught everywhere, and the exam stem often shows a leaderboard where the top-accuracy model is a majority-class predictor. Candidates who do not check class balance pick it automatically.
What to do instead
Before comparing models, ask what a false positive and a false negative cost in this scenario. Then read the leaderboard for the metric that matches that cost (recall when misses are expensive, precision when false alarms are expensive, F1 or cost-weighted scores when both matter). Treat a high-accuracy model on a rare-event target as a red flag and look for the option that changes the selection metric or the decision threshold.
Describing governance as documentation only: the candidate answers 'write a model card' or 'create a policy document' for scenarios that require enforced controls such as approval gates, access restrictions, or automated compliance checks.
Governance work in many organizations really is paperwork, so candidates map the exam's governance items onto the artifacts they have seen. The exam instead tests operating controls across the model lifecycle.
What to do instead
Organize governance answers by control point: intake and risk classification, data provenance and consent, approval before deployment, role-based access to models and data, change logging, periodic review, and retirement. When a scenario describes an unapproved model reaching production, the answer is a deployment gate with sign-off, not better documentation. Match the control to the lifecycle stage the stem describes.
Ignoring the difference between monitoring and observability: the candidate proposes dashboards and threshold alerts for a scenario about diagnosing an unexpected production failure, without mentioning logs, traces, and the ability to reconstruct a specific request.
Dashboards are the visible face of monitoring, so they come to mind first. Candidates miss that the exam is asking whether the system can be interrogated after the fact, not whether an alert fired.
What to do instead
Frame monitoring as watching known signals against thresholds and observability as the ability to ask new questions of production data. For incident-diagnosis scenarios, choose answers that preserve request-level traces, model inputs and outputs, retrieval context, latency breakdowns, and version identifiers. For early-warning scenarios, choose threshold and drift alerts. Let the verb in the stem ('detect' vs 'diagnose') pick the answer.
Missing the human-in-the-loop threshold: the candidate applies full automation or full manual review to every agentic or generative workflow instead of defining escalation criteria by risk and confidence.
Candidates tend to design for the average case and forget that the exam tests risk-tiered operations. Binary thinking about automation feels simpler and is easy to defend, so the nuanced option gets skipped.
What to do instead
For any workflow question, state the escalation rule: which actions are auto-approved, which require review above a risk or value threshold, which always require a human, and what confidence or policy signal triggers handoff. Then connect that rule to audit logging so every escalation is traceable. Answers that name a concrete threshold and a review path beat blanket automation or blanket manual review.
Treating prompt injection as a prompt-engineering problem: the candidate answers 'add instructions telling the model to ignore malicious input' rather than describing trust boundaries between instructions and untrusted content.
Prompting is the most accessible lever, so candidates reach for wording changes. The exam frames injection as a security and architecture issue, where instruction-level defenses are known to be insufficient.
What to do instead
Answer injection scenarios with structural controls: separate trusted instructions from untrusted retrieved or user content, constrain what tools an agent may call and with what permissions, validate and sanitize tool inputs and outputs, require confirmation for high-impact actions, and log every tool invocation. Mention that prompt-level defenses are a layer, not the boundary. Choose the option that limits blast radius over the one that adds a warning sentence.
Skipping the data-preparation stage in predictive answers: the candidate jumps to model selection and hyperparameters when the scenario describes leakage, missing values, or target imbalance introduced upstream.
Modeling is the most engaging part of the workflow and the part candidates study most. Data-quality items read as setup rather than as the actual question, so they get skimmed and answered with a modeling fix.
What to do instead
Read every predictive scenario for the data condition first: how the target was defined, whether features would be available at prediction time, how missing values were handled, and how the train/test split was made. If any of those is flawed, the correct answer addresses the data, not the algorithm. Practise spotting leakage by asking 'would this feature exist before the outcome occurred?' for each column mentioned.