DataRobot Certified Professional cheat sheet

An agent in DataRobot is composed of four parts: a model or LLM as the reasoning engine, tools (functions, APIs, or retrieval endpoints) it may call, a system prompt defining role and constraints, and a runtime that executes the plan-act-observe loop.

Tool definitions are JSON-schema-typed function signatures; the model selects a tool by name and arguments, and the platform validates arguments before execution, so an untyped or loosely typed tool is a common cause of failed agent runs.

Deterministic workflows are built as a graph of typed nodes (LLM step, tool step, conditional branch, human-approval step); the graph, not the model, governs control flow, which is what makes an agent auditable and testable.

Memory is separated into short-term conversation state (scoped to a session) and long-term retrievable knowledge (vector store); conflating the two causes context bloat and stale answers.

Retrieval-augmented generation in an agent is a tool call, not a prompt trick: retrieval returns chunks with scores, and the agent must be given a relevance threshold and a fallback instruction for when nothing clears it.

Guardrails are enforced at three chokepoints - input (prompt injection and PII screening), tool invocation (allow-list and argument bounds), and output (schema validation and content policy) - and each chokepoint should be tested independently.

Agent evaluation requires trace-level inspection: a run is scored on final answer correctness, tool-selection accuracy, argument validity, and step count, because a correct answer reached by a wrong tool path is not reproducible.

Human-in-the-loop approval nodes pause execution and persist state; the approval decision, approver identity, and timestamp become part of the run record and are the evidence artifact for governance review.

Generation parameters map directly to behavior: temperature and top-p control sampling randomness, max tokens caps output length and cost, and stop sequences terminate generation; lowering temperature does not fix factual error, retrieval does.

A vector store returns nearest neighbors by a similarity metric (cosine, dot product, or Euclidean); cosine and dot product rank identically only when embeddings are normalized, so normalization must match the metric configured.

Chunking decisions dominate retrieval quality: chunk size, overlap, and whether chunks respect document structure determine recall more than the choice of embedding model.

Hallucination mitigation is architectural, not prompt-only: grounding context, mandatory citation of retrieved chunk IDs, and an explicit refusal path when evidence is absent.

Prompt templates should be versioned artifacts with fixed system instructions and typed input variables; changing a template invalidates prior evaluation results and requires re-scoring.

Fine-tuning changes style, format adherence, and task behavior; it is a poor mechanism for injecting frequently changing factual knowledge, which belongs in retrieval.

LLM evaluation combines automated metrics (groundedness, answer relevance, toxicity) with rubric-based human or judge-model scoring; judge models must themselves be validated against a human-labeled sample.

Token cost is driven by prompt length times call count plus output length; long system prompts and multi-turn history are usually the largest hidden cost in a generative application.

Use-case type follows the target: regression for a continuous target, binary or multiclass classification for a categorical target, and time series for a target indexed by time with a forecast horizon.

Data partitioning rules: random or stratified splits for i.i.d. tabular data; time-aware or out-of-time splits whenever rows have temporal ordering, because random splits leak future information into training.

Target leakage is the most common cause of an implausibly strong model; any feature that would not be available at prediction time (or that encodes the outcome) must be removed before modeling.

The platform's leaderboard ranks models by a task-appropriate metric (for example logloss or AUC for classification, RMSE or MAE for regression); the best leaderboard score is not automatically the best production choice if it is unstable or uninterpretable.

Feature importance in tree ensembles should be read as a ranking aid, not causal evidence; use prediction explanations and partial dependence to describe direction and magnitude of effects.

Class imbalance is handled by choosing an appropriate metric (precision-recall AUC, F1) and by threshold tuning on the predicted probability, not by accuracy, which is misleading under imbalance.

A model's operating threshold is a business decision derived from the cost of false positives versus false negatives; the default 0.5 cut-off is rarely optimal.

Retraining triggers include data drift beyond a monitored threshold, degradation of a live performance metric once ground truth arrives, and scheduled refresh; retraining without re-validation risks regressing silently.

Governance artifacts attach to a specific model version, not to a project: documentation of training data, intended use, known limitations, and evaluation results must be reproducible for the exact deployed artifact.

Model documentation (model cards or equivalent) answers intended use, out-of-scope use, training data provenance, performance by segment, and known failure modes; missing segments are themselves a finding.

Bias assessment compares performance metrics across protected or sensitive groups; a fair overall metric can conceal large subgroup gaps, so disaggregated reporting is the standard.

Protected attributes require a documented decision: exclude from features, include only for fairness measurement, or include with justification; the choice and rationale must be recorded.

Access control and audit logging are governance controls: role-based permissions on projects and models, plus immutable logs of who trained, deployed, changed, or approved each artifact.

Approval workflows encode accountability - a named human approver, a timestamp, and a recorded rationale - and are the mechanism by which an organization demonstrates oversight.

Regulatory mapping is use-case dependent: high-risk classification triggers requirements for risk management, data governance, technical documentation, human oversight, and post-market monitoring under the EU AI Act.

Model retirement and decommissioning are governance events requiring notification to consumers of the model output and archival of the final documented state.

Data drift measures change in the distribution of input features versus a training or reference baseline; it can be detected without labels and is the earliest warning signal.

Concept drift is change in the relationship between features and target; it cannot be confirmed without ground truth and shows up as degradation in live accuracy, AUC, or RMSE.

Prediction drift tracks shifts in the model's own output distribution - score mean, variance, and positive-rate - and often precedes measured accuracy loss.

Service health monitoring covers latency percentiles, error and timeout rates, and throughput; a healthy model behind a failing endpoint is an outage, so these are first-class signals.

Ground-truth joining is the prerequisite for performance monitoring: predictions must be stored with an identifier and joined to outcomes when they arrive, with the lag documented.

Alert thresholds should be set from baseline variability rather than round numbers; a threshold that fires on normal fluctuation produces alert fatigue and is ignored.

Champion–challenger deployment compares a live production model against a candidate on the same traffic, enabling evaluation of the challenger without exposing all users to it.

Incident response for a degraded model includes rollback to the previous version, root-cause classification (data pipeline, upstream schema change, genuine world change), and a recorded post-incident review.

Define the agent's tool contract before selecting a model.

Reasoning engines are swappable; tool schemas and graph topology are the durable design.

Every agent run should emit a trace with steps, tool calls, and token counts.

Approval nodes convert agent autonomy into an auditable human decision.

Retrieval failures usually trace to chunking, not to the model.

Guardrail tests belong in the same suite as functional tests.

A correct answer via an invalid tool path fails reproducibility review.

Session state and knowledge store must be configured separately.

Temperature and top-p control randomness, not correctness.

Cosine similarity requires normalized vectors to match dot product ranking.

Chunk size and overlap are the highest-leverage retrieval settings.

Require chunk-ID citations to make groundedness checkable.

Version prompt templates; any edit invalidates prior evaluations.

Use retrieval for changing facts and fine-tuning for behavior.

Judge-model scores are valid only after calibration against human labels.

Prompt length times call count dominates generative cost.

Time-ordered data requires out-of-time validation splits.

Remove any feature unavailable at prediction time to prevent leakage.

Leaderboard rank is a starting filter, not the final selection.

Choose metrics that survive class imbalance: PR-AUC, F1, recall.

Tune the decision threshold to the false-positive/false-negative cost ratio.

Feature importance ranks; prediction explanations show direction.

Retrain on drift thresholds or confirmed performance decay.

Re-validate every retrained model before promotion.

Documentation binds to a model version, never to a project.

Report performance disaggregated by sensitive group.

Record the decision and rationale for each protected attribute.

Approvals need a named approver, timestamp, and rationale.

Immutable audit logs cover train, deploy, change, and approve events.

High-risk use cases trigger documentation and oversight duties.

Retirement is a governed event with consumer notification.

Missing subgroup data is a finding, not a neutral gap.

Data drift is detectable without labels; concept drift is not.

Prediction drift often precedes measurable accuracy loss.

Ground-truth lag must be documented before setting perf alerts.

Derive alert thresholds from baseline variability, not round numbers.

Champion–challenger isolates challenger risk on shared traffic.

Latency and error rates are model health signals, not infra trivia.

Store prediction IDs to enable later outcome joins.

Rollback is the first response to confirmed degradation.