I85PREP
⌘ K
DataRobot Certified Professional study materials
DataRobot Certified Professional study resources

DataRobot Certified Professional practice questions

One question at a time. Build your DataRobot Certified Professional understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Agentic AI Platform

In the DataRobot agentic AI platform, what is the primary purpose of a tool (also called a tool definition) exposed to an agent?

  1. To schedule periodic retraining of the underlying predictive models
  2. To convert the agent's natural language output into a deployed prediction API
  3. To describe a callable capability with a name, inputs, and outputs so the agent can invoke it Correct answer.
  4. To store the agent's long-term conversation history between sessions

Explanation

Tools are the contract through which an agent reaches external capabilities: the platform registers a name, a description, and a typed input/output schema, and the agent selects the tool at runtime. The description is what the model reasons over when deciding which capability fits the user request.

Question 2Agentic AI Platform

A team wants an agent to answer questions strictly from an approved internal policy corpus and cite the passages used. Which platform capability should they configure first?

  1. A retrieval-augmented generation pipeline backed by a vectorized knowledge base Correct answer.
  2. A guard model that blocks profanity in the agent's responses
  3. A code execution tool that lets the agent run Python
  4. A prompt template that instructs the agent to be concise

Explanation

Grounding answers in an approved corpus requires retrieval: documents are chunked, embedded, and stored so the agent can retrieve relevant passages at query time and cite them. Guard models and prompt templates shape behavior but do not supply the source content.

Question 3Agentic AI Platform

What distinguishes an agentic workflow from a single-shot LLM prompt in the DataRobot platform?

  1. The output is passed through a content moderation filter
  2. The response is generated with a lower temperature setting
  3. The model plans and selects among tools across multiple steps to reach a goal Correct answer.
  4. The prompt is stored in a version-controlled template registry

Explanation

Agentic behavior means the system decides what to do next, choosing tools and iterating over several steps rather than producing one completion. Temperature, template storage, and moderation are configuration details that can apply to either mode.

Question 4Agentic AI Platform

Which component determines whether an agent may call a specific tool in a production deployment?

  1. The latency of the tool's underlying endpoint
  2. The token count of the tool's description string
  3. The tool's permission and access policy configured for the agent Correct answer.
  4. The number of examples in the tool's few-shot prompt

Explanation

Access to tools is governed by explicit permissions bound to the agent or its runtime identity, so administrators can restrict sensitive capabilities. Description length, latency, and prompt examples affect performance or cost but not authorization.

Question 5Agentic AI Platform

An agent must remember a user's stated preferences across separate conversations. Which mechanism is designed for this?

  1. Persistent memory scoped to the user identity Correct answer.
  2. The tool call timeout setting
  3. The model's context window for the current request
  4. The agent's system prompt

Explanation

Long-term memory is stored outside the immediate request and keyed to an identity such as the user, so it survives across sessions. The context window only holds what is assembled for the current call, and the system prompt is static configuration.

Question 6Agentic AI Platform

What is the role of an agent's system prompt in the DataRobot agentic platform?

  1. It sets persistent instructions such as role, constraints, and tone for the agent Correct answer.
  2. It stores the retrieved documents used to ground each answer
  3. It defines the schema of every tool the agent can call
  4. It records each tool invocation for audit purposes

Explanation

The system prompt carries standing instructions that apply to every turn, shaping persona, boundaries, and style. Tool schemas, retrieval content, and audit logs live in their own configuration and telemetry layers.

Question 7Agentic AI Platform

A developer wants the agent to hand off a complex billing dispute to a human queue. Which platform concept supports this?

  1. A higher temperature setting for the model
  2. A larger context window for the model
  3. A second retrieval index over billing documents
  4. An escalation or handoff action that transfers the conversation with context Correct answer.

Explanation

Human-in-the-loop handoff is implemented as an explicit escalation action that packages the conversation state for a human agent. Model parameters and additional indexes change how the agent answers but do not transfer control to a person.

Question 8Agentic AI Platform

Which statement best describes how the DataRobot platform treats an agent deployment endpoint?

  1. It bypasses platform governance because agents are non-deterministic
  2. It exposes the configured agent as a governed service with monitoring and access control Correct answer.
  3. It requires the agent to be converted into a traditional regression model
  4. It disables logging to protect conversational privacy

Explanation

Agent deployments are managed endpoints subject to the same governance posture as other deployments, including authentication, monitoring, and logging. Non-determinism is handled through observability rather than by removing controls.

Question 9Agentic AI Platform

An agent repeatedly calls a search tool with malformed arguments, causing failures. Which change most directly improves reliability?

  1. Tighten the tool's input schema and description so required fields and formats are explicit Correct answer.
  2. Lower the agent's temperature to zero and change nothing else
  3. Increase the model's maximum output tokens
  4. Add more retrieved documents to the knowledge base

Explanation

Malformed arguments usually indicate an ambiguous tool contract, so clarifying the schema and description gives the model the constraints it needs to construct valid calls. Output length, retrieval content, and sampling settings do not fix argument structure.

Question 10Agentic AI Platform

A retrieval-grounded agent answers correctly in testing but cites irrelevant passages in production. Which adjustment is most appropriate?

  1. Remove the citation instruction from the system prompt
  2. Disable retrieval and rely on the model's parametric knowledge
  3. Raise the model temperature to encourage diverse citations
  4. Tune chunk size and retrieval top-k, then evaluate groundedness on a labeled set Correct answer.

Explanation

Citation quality depends on whether the retrieved chunks actually contain the supporting evidence, so chunking and retrieval depth should be tuned and measured against labeled examples. Raising temperature or dropping retrieval would worsen grounding.

Question 11Agentic AI Platform

During an agent run, a tool returns a transient timeout. What is the most robust design for the agent's tool layer?

  1. Silently substitute a plausible fabricated result
  2. Retry indefinitely until the tool eventually responds
  3. Fail the entire conversation with no diagnostic message
  4. Retry with bounded backoff and surface a clear failure to the agent if retries are exhausted Correct answer.

Explanation

Bounded retries with backoff handle transient faults without hanging the run, and an explicit failure lets the agent explain the limitation or choose another path. Infinite retries and fabrication create worse outcomes than a clean, informative failure.

Question 12Agentic AI Platform

A team needs the agent to answer only from a specific set of documents and refuse otherwise. Which combination implements this most directly?

  1. A code execution tool plus a summarization prompt
  2. Retrieval over the approved corpus plus a grounded-answer instruction with an abstain path Correct answer.
  3. A larger model plus a longer maximum response length
  4. A higher top-k over a general web search index

Explanation

Restricting the knowledge source and instructing the agent to abstain when evidence is absent creates a closed-book boundary. Model size, code execution, and broader web retrieval all widen the answer space rather than constrain it.

Question 13Agentic AI Platform

An agent must complete a multi-step task: look up a customer, check order status, then draft a reply. Which design keeps the flow reliable and auditable?

  1. Let the agent call a single generic HTTP tool with arbitrary URLs
  2. Combine all three steps into one large free-text prompt with no tools
  3. Rely on the model's memory of prior sessions to supply order data
  4. Define discrete tools per step and log each invocation with its inputs and outputs Correct answer.

Explanation

Separate, well-scoped tools make each step observable and let the platform record inputs and outputs for audit and debugging. A single generic tool or one unstructured prompt hides intermediate state and weakens control.

Question 14Agentic AI Platform

What is the main benefit of evaluating an agent with a scenario-based test suite rather than single-turn prompts?

  1. It removes the need for production monitoring
  2. It guarantees deterministic outputs for every run
  3. It eliminates the possibility of prompt injection
  4. It exercises planning, tool selection, and recovery across multiple turns Correct answer.

Explanation

Agent quality emerges over a trajectory, so scenario tests check whether the agent plans, picks appropriate tools, and recovers from errors across turns. Such tests cannot make a generative system deterministic or replace live monitoring.

Question 15Agentic AI Platform

A prompt injection attempt arrives inside a retrieved document. Which control most directly limits its impact?

  1. Use a higher temperature so responses vary
  2. Shorten the system prompt to reduce token cost
  3. Treat retrieved content as untrusted data and restrict the tools the agent may invoke Correct answer.
  4. Increase the model's context window to include more documents

Explanation

Indirect injection works when untrusted content can influence privileged actions, so separating data from instructions and least-privilege tool access contains the blast radius. Context size, sampling, and prompt length do not address the trust boundary.

Question 16Agentic AI Platform

Which metric best captures whether an agent's final answer is supported by the sources it retrieved?

  1. Tool call count per conversation
  2. Average response latency in milliseconds
  3. Model parameter count
  4. Groundedness, scored by checking claims against the cited passages Correct answer.

Explanation

Groundedness measures whether each claim in the answer is entailed by the retrieved evidence, which is exactly the property citation-based answering must satisfy. Latency, call counts, and model size describe performance or scale, not evidential support.

Question 17Agentic AI Platform

An agent's memory contains an outdated shipping preference from six months ago. What is the best governance practice?

  1. Store all memories permanently to maximize personalization
  2. Apply time-based expiry or confidence rules so stale memories can be superseded Correct answer.
  3. Disable memory entirely for every user
  4. Copy the memory into the system prompt so it is always applied

Explanation

Memories should carry freshness and confidence so outdated facts can be retired or overridden by newer statements. Permanent retention risks acting on stale data, while disabling memory or hard-coding it in the prompt sacrifices flexibility and control.

Question 18Agentic AI Platform

Which configuration most directly reduces an agent's risk of taking an irreversible action such as issuing a refund?

  1. Increase the model's temperature for more cautious wording
  2. Require human approval for high-impact tools before execution Correct answer.
  3. Enable a longer conversation history window
  4. Add more examples of refunds to the system prompt

Explanation

High-impact actions should pass through an approval gate so a human confirms before execution, which is the standard human-in-the-loop control. Wording, examples, and history length influence behavior but do not prevent an irreversible call.

Question 19Agentic AI Platform

A team observes that the agent ignores a newly added tool. What is the most likely cause?

  1. The agent's memory is enabled
  2. The deployment has logging turned on
  3. The tool's description does not clearly state when it should be used Correct answer.
  4. The tool's endpoint has low latency

Explanation

Tool selection is driven by the description, so a vague or overlapping description leaves the model unable to recognize when the tool applies. Latency, memory, and logging do not determine whether the model chooses a tool.

Question 20Agentic AI Platform

Which practice best supports auditing why an agent produced a particular answer in production?

  1. Store only the final response text
  2. Rely on the model provider's aggregate usage dashboard
  3. Persist a trace linking the prompt, retrieved passages, tool calls, and final response Correct answer.
  4. Record only the total token count per request

Explanation

An end-to-end trace connects the inputs, evidence, and actions that led to an output, enabling root-cause analysis and compliance review. Response-only or aggregate records cannot reconstruct the reasoning path.

Question 21Agentic AI Platform

An agent must choose between two tools with overlapping descriptions. What is the best way to make selection predictable?

  1. Set the temperature to its maximum value
  2. Increase the number of tools available to give the model more options
  3. Remove descriptions so the model relies on tool names alone
  4. Differentiate the descriptions by clearly stating each tool's distinct use case and boundaries Correct answer.

Explanation

When descriptions overlap, the model has no reliable signal for choosing, so sharpening each description's scope and boundaries resolves ambiguity. Adding tools or stripping descriptions increases confusion, and high temperature adds variability.

Question 22Agentic AI Platform

A regulated team requires that no customer PII reach a third-party model provider. Which architecture satisfies this while preserving agent capability?

  1. Send full records but shorten the system prompt
  2. Apply redaction or a PII guard before external model calls and keep sensitive tools internal Correct answer.
  3. Log the PII so it can be reviewed after the fact
  4. Rely on the provider's terms of service to prevent data retention

Explanation

Preventing PII from leaving the trust boundary requires detecting and redacting it before external calls, with sensitive operations handled by internal tools. Policy reliance and post-hoc logging do not stop the data from being transmitted.

Question 23Agentic AI Platform

An agent's tool calls succeed in isolation but the overall task fails because it stops before the final step. Which failure mode should be investigated first?

  1. The deployment endpoint's TLS certificate
  2. The retrieval index's embedding dimension
  3. The model's parameter count relative to the provider's maximum
  4. Planning and termination logic, including stopping criteria in the agent loop Correct answer.

Explanation

Premature stopping points to the agent loop's planning and completion criteria, such as an instruction or condition that treats partial progress as done. Retrieval dimensions, transport security, and model size would not selectively truncate a multi-step plan.

Question 24Agentic AI Platform

A team wants to compare two agent configurations fairly on cost and quality. Which evaluation design is most defensible?

  1. Choose the configuration with the larger underlying model
  2. Run both on the same labeled scenarios and compare quality scores alongside token and tool-call costs Correct answer.
  3. Compare only the average latency of each configuration
  4. Compare the number of prompt templates each configuration uses

Explanation

A fair comparison holds the test set constant and reports quality and cost together, since an agent can look cheaper only by doing less work. Latency, template counts, and model size are proxies that do not capture task success.

Question 25Agentic AI Platform

An agent has permission to call a write tool that modifies CRM records. Which control most reduces the risk of unintended bulk changes?

  1. Increase the agent's context window so it sees more records
  2. Remove logging on write calls to reduce storage cost
  3. Grant broad write access but rely on the model's judgment
  4. Scope the tool to single-record operations with validated identifiers and rate limits Correct answer.

Explanation

Least-privilege tool design limits each call to one validated record and caps call frequency, containing the impact of a mistaken plan. Broad access, reduced logging, and larger context all increase exposure without adding control.

Question 26Agentic AI Platform

In the DataRobot agentic platform, which artifact defines the instructions and behavioral boundaries an agent follows on every turn?

  1. The feature drift monitor
  2. The deployment's prediction threshold
  3. The system prompt attached to the agent configuration Correct answer.
  4. The model retraining schedule

Explanation

An agent's behavior is governed by its system prompt, which is part of the agent configuration and is applied to every turn of a conversation. Deployment thresholds, drift monitors, and retraining schedules belong to classical model operations and do not define conversational instructions.

Question 27Agentic AI Platform

A support agent must escalate unresolved cases to a human representative with full conversation context. Which platform concept should the designer configure?

  1. A human-in-the-loop handoff that transfers the session and its transcript Correct answer.
  2. A scheduled batch scoring job on the ticket table
  3. A higher temperature setting on the underlying model
  4. An additional retrieval collection of past tickets

Explanation

Handoff is the mechanism that transfers an in-progress agent session to a human queue while preserving context so the representative can continue without restarting the conversation. Temperature, extra retrieval collections, and batch scoring affect model output or offline processing, not escalation routing.

Question 28Agentic AI Platform

What does a DataRobot agent deployment endpoint primarily expose to client applications?

  1. A callable interface that accepts a request and returns the agent's response for that session Correct answer.
  2. A file share containing raw prompt templates
  3. A read-only dashboard of historical training metrics
  4. A direct SQL connection to the underlying feature store

Explanation

An agent deployment endpoint is the runtime interface clients call to send input and receive agent responses, with session context handled by the platform. It is not a metrics dashboard, a database connection, or a raw file location.

Question 29Agentic AI Platform

An agent fails intermittently because a search tool sometimes returns malformed argument errors. Which change most directly improves reliability?

  1. Shorten the system prompt by removing examples
  2. Increase the agent's maximum token budget
  3. Disable logging on the tool to reduce overhead
  4. Tighten the tool's input schema and add validation with a corrective retry message Correct answer.

Explanation

Malformed tool arguments are best addressed by a strict input schema plus validation that returns a corrective message the agent can act on before retrying. Token budget, prompt length, and logging settings do not correct the argument mismatch itself.

Question 30Agentic AI Platform

A retrieval-grounded agent gives accurate answers in testing but cites unrelated passages in production. Which adjustment should be investigated first?

  1. The frequency of scheduled model retraining
  2. The agent's conversation memory retention window
  3. The deployment's autoscaling replica count
  4. The retrieval configuration and chunking used in the production knowledge source Correct answer.

Explanation

Citation quality depends on which passages retrieval returns, so production retrieval configuration and chunking are the first suspects when citations stop matching answers. Memory windows, replica counts, and retraining cadence do not determine which source passages are retrieved.

Question 31Agentic AI Platform

A tool call in an agent run returns a transient timeout. Which tool-layer design is most robust?

  1. Silent substitution of a fabricated result
  2. Permanent disablement of the tool after one failure
  3. Bounded retries with backoff, then a defined fallback response Correct answer.
  4. Immediate retry in an unbounded loop until success

Explanation

Transient failures should be absorbed by a bounded retry policy with backoff and an explicit fallback so the agent degrades gracefully instead of looping or inventing data. Unbounded retries risk runaway cost, fabrication breaks trust, and permanent disablement overreacts to a temporary condition.

Question 32Agentic AI Platform

A team needs an agent to answer only from an approved document set and decline otherwise. Which combination implements that requirement?

  1. A longer system prompt plus more few-shot examples
  2. A larger context window plus a lower temperature
  3. A higher request rate limit plus response caching
  4. Retrieval restricted to the approved collection plus an instruction to decline when evidence is insufficient Correct answer.

Explanation

Scoping retrieval to the approved collection limits the evidence available, and an explicit insufficiency instruction tells the agent to refuse when that evidence is absent. Context size, temperature, prompt length, and rate limits do not constrain the knowledge boundary.

Question 33Agentic AI Platform

An agent must look up a customer, check order status, then draft a reply. Which design keeps the multi-step task coherent?

  1. Parallel invocation of both tools with no ordering
  2. Randomized tool selection on each turn
  3. A single prompt asking the model to do everything at once
  4. An explicit plan with ordered steps and state passed between tool calls Correct answer.

Explanation

Multi-step tasks stay coherent when the agent maintains an explicit plan and carries state such as the customer identifier from one tool call to the next. Collapsing everything into one prompt, removing ordering, or randomizing selection breaks the dependency chain.

Question 34Agentic AI Platform

Why is a scenario-based test suite more informative than isolated single-turn prompts when evaluating an agent?

  1. It guarantees identical outputs across model versions
  2. It eliminates the need for any human review of outputs
  3. It measures only latency rather than answer quality
  4. It exercises multi-turn tool use and state handling that single prompts never reach Correct answer.

Explanation

Agents fail in ways that emerge across turns, such as losing state or misusing tools, so scenario suites reveal behavior that single prompts cannot surface. They do not remove human review, guarantee determinism, or limit evaluation to latency.

Question 35Agentic AI Platform

A retrieved document contains text instructing the agent to ignore its rules. Which control most directly limits the impact?

  1. Increase the model's temperature so responses vary more
  2. Add the retrieved document to long-term memory
  3. Treat retrieved content as untrusted data and keep instructions confined to the system prompt Correct answer.
  4. Allow the agent to execute commands found in documents

Explanation

Prompt injection is mitigated by establishing a trust boundary where retrieved text is data, never instructions, and only the system prompt carries directives. Raising temperature, persisting the malicious text, or executing embedded commands increases exposure.

Question 36Agentic AI Platform

Which metric most directly measures whether an agent's answer is supported by the sources it retrieved?

  1. Tool call count per session
  2. Groundedness or faithfulness of the response to retrieved evidence Correct answer.
  3. Number of active deployment replicas
  4. Average response latency

Explanation

Groundedness, also called faithfulness, assesses whether claims in the answer are entailed by the retrieved evidence. Latency, tool call counts, and replica counts describe operational behavior rather than evidential support.

Question 37Agentic AI Platform

An agent's memory holds a shipping preference recorded six months ago that the customer has since changed. What is the best governance practice?

  1. Store the preference without any timestamp or provenance
  2. Apply time-based expiry and re-confirmation so stale preferences are refreshed before use Correct answer.
  3. Retain the preference indefinitely to preserve conversational continuity
  4. Let the model decide silently whether the preference is current

Explanation

Memory entries should carry timestamps and provenance and be subject to expiry or re-confirmation so outdated facts are not applied as if current. Indefinite retention, missing metadata, and silent model judgment all allow stale data to influence decisions.

Question 38Agentic AI Platform

Which configuration most directly reduces the risk of an agent issuing an irreversible refund on its own?

  1. Adding more refund examples to the system prompt
  2. Enabling verbose logging of every tool call
  3. Raising the agent's maximum reasoning steps
  4. Requiring human approval before the refund tool executes Correct answer.

Explanation

Irreversible actions are controlled by gating the write tool behind an approval step so execution cannot occur without human sign-off. More reasoning steps, additional examples, and verbose logging change behavior or observability but do not block the action.

Question 39Agentic AI Platform

An agent ignores a newly registered tool even though the tool works when called directly. What is the most likely cause?

  1. The tool's server has too many replicas
  2. The knowledge collection contains too many documents
  3. The tool's description and schema were not made available in the agent's tool context Correct answer.
  4. The deployment's autoscaling policy is misconfigured

Explanation

Agents select tools from the descriptions and schemas exposed in their context, so a tool that is registered but absent from that context will not be chosen. Replica counts, autoscaling, and collection size do not determine tool visibility to the model.

Question 40Agentic AI Platform

A regulated team needs to compare two agent configurations fairly on quality and cost. Which evaluation design is most defensible?

  1. Evaluate each configuration on a different scenario set chosen by its own author
  2. Compare total token spend without measuring answer quality
  3. Compare only the cheaper configuration's outputs against itself over time
  4. Run both configurations on the same fixed scenario set with identical tool access and compare quality and cost together Correct answer.

Explanation

A fair comparison holds inputs and capabilities constant, so both configurations should face the same scenarios with equivalent tool access while quality and cost are measured jointly. Different scenario sets, self-comparison, or cost-only views introduce bias or ignore quality.

Question 41Generative AI

In DataRobot's generative AI offering, what is the primary role of a vector database in a retrieval-augmented generation (RAG) pipeline?

  1. It enforces content moderation policies on generated outputs
  2. It stores embeddings of source documents and returns the nearest chunks to a query embedding Correct answer.
  3. It caches the model's tokenizer vocabulary to reduce inference latency
  4. It fine-tunes the foundation model weights on each incoming user request

Explanation

In a RAG architecture the vector database indexes embedded document chunks so that at query time the system can retrieve semantically nearest chunks to ground the model's answer. It does not train or alter model weights, and moderation and tokenization are handled by other components.

Question 42Generative AI

A team wants the LLM in their DataRobot generative AI application to answer questions using internal policy PDFs that change weekly. Which approach best keeps answers current without retraining the model?

  1. Lower the maximum token limit on the completion to force brevity
  2. Switch to a smaller model with fewer parameters to reduce cost
  3. Ingest the updated PDFs into the retrieval corpus so relevant chunks are fetched at query time Correct answer.
  4. Increase the model's temperature setting so it explores more answers

Explanation

Retrieval-augmented generation grounds answers in an external corpus, so refreshing the indexed documents updates the knowledge base without any model retraining. Temperature, token limits, and model size affect style or cost, not factual currency.

Question 43Generative AI

Which statement accurately describes an embedding as used in DataRobot generative AI workflows?

  1. A human-readable summary generated for each document chunk
  2. A numeric vector representing semantic meaning, positioned so similar texts are close together Correct answer.
  3. A cryptographic hash that uniquely identifies each source document
  4. A compressed archive of the original document used for storage savings

Explanation

Embeddings are dense numeric vectors produced by an embedding model; texts with similar meaning map to nearby points, enabling semantic search. They are not summaries, archives, or hashes, though hashes are sometimes used for deduplication.

Question 44Generative AI

What does chunking refer to when preparing documents for a DataRobot RAG application?

  1. Batching multiple user prompts into a single API call
  2. Dividing the model's context window among concurrent users
  3. Grouping evaluation prompts by difficulty tier
  4. Splitting source documents into smaller passages that are embedded and retrieved individually Correct answer.

Explanation

Chunking divides long documents into passages sized to fit retrieval and context limits, and each chunk is embedded separately. Good chunking balances semantic completeness against retrieval precision.

Question 45Generative AI

In DataRobot's generative AI guardrails, what is the purpose of prompt injection detection?

  1. To measure how many tokens each prompt consumes for billing
  2. To identify inputs that attempt to override system instructions or exfiltrate hidden context Correct answer.
  3. To route requests to the fastest available model endpoint
  4. To translate user prompts into the model's training language

Explanation

Prompt injection detection flags adversarial inputs designed to make the model ignore its instructions or leak system prompts and retrieved data. It is a security control, distinct from metering, translation, or routing.

Question 46Generative AI

A practitioner needs to compare two candidate LLMs on a fixed set of question-answer pairs in DataRobot. Which capability should they use?

  1. A feature importance report from the underlying classifier
  2. A drift monitor configured on the embedding store
  3. A batch scoring job on the training dataset
  4. A model comparison evaluation that runs both models against the same test cases and scores the outputs Correct answer.

Explanation

Side-by-side evaluation runs candidate models over a shared evaluation dataset and scores outputs with chosen metrics or judge models, enabling fair comparison. Training-time reports and drift monitors do not compare generative outputs on test cases.

Question 47Generative AI

Which metric is most appropriate for judging a RAG system's retriever independently of the generator?

  1. BLEU score of the final answer
  2. Recall at k of the relevant chunks in the retrieved set Correct answer.
  3. Latency of the embedding model
  4. Perplexity of the generated completion

Explanation

Retrieval quality is measured by whether relevant chunks appear among the top-k results, commonly reported as recall at k or similar ranking metrics. BLEU and perplexity assess generated text, and latency measures speed rather than relevance.

Question 48Generative AI

What is hallucination in the context of a DataRobot generative AI application?

  1. The model failing to respond because of a rate limit
  2. The vector database returning too many chunks for the context window
  3. The application returning a cached response instead of a fresh one
  4. The model producing fluent output that is unsupported by or contradicts the provided source context Correct answer.

Explanation

Hallucination is confident but fabricated or unsupported content, which is why grounding via retrieval and faithfulness checks matter. Rate limits, oversized context, and caching are operational issues rather than hallucination.

Question 49Generative AI

A RAG application gives correct answers for simple questions but misses answers when the relevant passage spans two adjacent chunks. What is the most targeted fix?

  1. Raise the temperature to encourage broader answers
  2. Reduce the number of retrieved chunks to one
  3. Increase chunk overlap or use larger chunks with overlap so boundary content is preserved Correct answer.
  4. Remove the reranker to speed up retrieval

Explanation

Content split across chunk boundaries can be lost when each chunk is embedded alone; adding overlap or larger chunks preserves cross-boundary context. Retrieving fewer chunks or removing reranking would worsen recall.

Question 50Generative AI

What is the function of a reranker in a DataRobot RAG pipeline?

  1. To retrain the embedding model on the user's query
  2. To regenerate the answer a second time and pick the longer output
  3. To compress the retrieved chunks into a single sentence
  4. To reorder retrieved candidates by relevance to the query before they are passed to the LLM Correct answer.

Explanation

A reranker scores the initially retrieved candidates more precisely against the query and reorders them so the most relevant context reaches the generator. It does not retrain embeddings or select outputs by length.