DataRobot Certified Professional flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Agent tool schema

A typed function signature, usually JSON Schema, that declares a tool's name, parameters, and types so the runtime can validate model-generated arguments before execution.

Plan-act-observe loop

The agent runtime cycle in which the model plans, invokes a tool, receives the observation, and re-plans until it produces a final answer or hits a step limit.

Human-in-the-loop approval node

A workflow step that pauses agent execution, persists state, and requires a named human decision whose identity, timestamp, and rationale are recorded in the run trace.

Agent trace evaluation

Scoring a full run on final answer correctness, tool-selection accuracy, argument validity, and step count, so that a right answer reached by a wrong path is caught.

Agent guardrail chokepoints

Three enforcement points: input screening for injection and PII, tool invocation allow-listing and argument bounds, and output schema and content-policy validation.

Session versus knowledge memory

Session memory holds conversation state scoped to one interaction; knowledge memory is a persistent vector store retrieved on demand. Mixing them causes context bloat.

Deterministic agent workflow graph

A typed node graph of LLM steps, tool calls, branches, and approvals where the graph rather than the model governs control flow, making behavior testable and auditable.

Retrieval as a tool call

Treating retrieval as an invocable tool that returns scored chunks, with a configured relevance threshold and an explicit fallback instruction when no chunk clears it.

Temperature and top-p

Sampling parameters controlling output randomness. Lower temperature makes output more deterministic but does not correct factual errors, which require grounding or retrieval.

Cosine versus dot product

Two vector similarity metrics that rank neighbors identically only when embeddings are normalized; a mismatch between stored normalization and configured metric distorts retrieval.

Chunking strategy

Decisions about chunk size, overlap, and structure awareness that determine retrieval recall more strongly than the choice of embedding model.

Groundedness metric

An evaluation measure of whether generated claims are supported by the supplied context, typically computed by requiring citations to retrieved chunk identifiers.

Prompt template versioning

Treating prompts as versioned artifacts with fixed system instructions and typed variables, so that any edit invalidates prior evaluation results and forces re-scoring.

Fine-tuning versus retrieval

Fine-tuning changes style, format adherence, and task behavior; retrieval injects frequently changing factual knowledge. Using fine-tuning for facts produces stale answers.

Judge model calibration

Validating an LLM used as an automated evaluator against a human-labeled sample before trusting its scores, since uncalibrated judges show systematic bias.

Generative token cost driver

Cost equals prompt tokens times call count plus output tokens; long system prompts and retained conversation history are typically the largest hidden contributors.

Out-of-time validation split

A train/test partition by time where the test set is strictly later than training data, preventing future information from leaking into the model.

Target leakage

Including a feature that encodes the outcome or is unavailable at prediction time, producing implausibly strong offline metrics that collapse in production.

Leaderboard selection caveat

The top-ranked model by a task metric is a starting filter, not the final choice; stability, interpretability, and latency can justify a lower-ranked model.

Class imbalance metrics

Under imbalance, accuracy misleads; use precision-recall AUC, F1, or recall at a chosen threshold, and tune the probability cut-off rather than accepting 0.5.

Decision threshold economics

The operating cut-off on predicted probability is set from the relative cost of false positives versus false negatives, making it a business rather than statistical decision.

Feature importance interpretation

Tree-ensemble importance ranks features by contribution but is not causal evidence; prediction explanations and partial dependence show direction and magnitude.

Retraining triggers

Conditions that initiate retraining: data drift past a monitored threshold, confirmed live metric degradation once ground truth arrives, or a scheduled refresh cycle.

Time series use case

A modeling task where the target is indexed by time and predictions are required over a forecast horizon, requiring time-aware validation and horizon-specific metrics.

Version-bound model documentation

Governance records covering intended use, out-of-scope use, data provenance, segment performance, and limitations, attached to one specific deployed model version.

Disaggregated bias assessment

Comparing performance metrics across protected or sensitive groups, because an acceptable overall metric can conceal large subgroup performance gaps.

Protected attribute decision

A recorded choice to exclude a sensitive attribute from features, retain it only for fairness measurement, or include it with documented justification.

Governance audit logging

Immutable records of who trained, deployed, modified, or approved each model artifact, paired with role-based access control over projects and models.

High-risk regulatory triggers

Use-case classifications that impose risk management, data governance, technical documentation, human oversight, and post-market monitoring obligations under the EU AI Act.

Data drift

Change in the distribution of input features relative to a training or reference baseline; detectable without labels and the earliest available warning signal.

Concept drift

Change in the relationship between features and target; it cannot be confirmed without ground truth and appears as degradation in live accuracy, AUC, or RMSE.

Champion-challenger deployment

Running a candidate model alongside the production champion on shared traffic so the challenger can be evaluated without exposing all users to its predictions.