Databricks GenAI Engineer common mistakes
Databricks GenAI Engineer · Understand the traps and how to avoid them.
Selects a serving option by name recognition alone: points the app at a pay-per-token Foundation Model API endpoint, a provisioned throughput endpoint, or an external model endpoint without verifying which one supports the chosen model, request schema, context length, or throughput - then ships code with a payload format copied from a different provider and no rate-limit handling.
Candidates memorise 'use Foundation Model APIs' as a slogan and never learn that endpoint type determines supported models (e.g. whether a fine-tuned or non-Databricks-hosted model is served), quota behaviour, and JSON contract. The first 429 or 400 schema error appears only at demo time.
What to do instead
For each design question, write down the requirement first (latency SLA, throughput, custom/fine-tuned weights, data residency), then map it to pay-per-token, provisioned throughput, or external model serving. Practice the actual request/response JSON for each, including retry-with-backoff and fallback-endpoint logic, and check the supported-model list before committing to an endpoint name.
Stuffs a fixed top-k of retrieved chunks (or the full conversation history) into every prompt with no token budgeting, no chunk-size cap, and no structured output contract, so prompts silently exceed the model context window, get truncated, or return free text the downstream code cannot parse.
Prototyping with one small document makes context limits invisible, and candidates treat 'more context = better answers'. Truncation and parse failures look like model quality problems rather than engineering ones.
What to do instead
Size the prompt explicitly: reserve tokens for system prompt, output, and history; rank and truncate chunks to fit; summarise or window long conversations. Define the output contract (JSON schema / function-calling tool definition) and validate the response before downstream use, with a repair or retry path.
Chunks documents by fixed character or paragraph counts with no overlap, no token awareness, no respect for structural boundaries (tables, headers, code blocks), and throws away source metadata such as file path, section, page, and document ID.
Character counts are easy to compute and 'feels like chunking', but embedding models have token limits and answers frequently span a sentence or table that got cut in half. Metadata loss is invisible until you need citations or filtering.
What to do instead
Parse first (proper document parsing for PDFs/tables), chunk on structural boundaries with token-based sizing and a 10-20% overlap, and store metadata (source, section, page, timestamps, access-control tags) alongside each chunk so retrieval can filter and answers can cite.
Builds the Vector Search index once and never manages its lifecycle: no sync triggered after the source Delta table changes, confusion between Delta Sync and Direct Vector Access modes, and re-embedding with a different or upgraded embedding model while queries still use the old query-side model or dimension.
The index looks correct on the demo dataset. Staleness, sync-mode mismatch, and embedding-model drift produce silently wrong retrieval rather than errors, so nobody notices until a user complains about outdated answers.
What to do instead
Decide deliberately between Delta Sync (managed sync of a source table, including continuous or triggered sync) and Direct Vector Access (you push vectors), and set up the sync trigger as part of the pipeline. Treat the embedding model as a versioned contract: pin model and dimensions, record them in the index, and rebuild fully whenever either the model or the chunking strategy changes.
Reaches for fine-tuning as the default fix for 'the model does not know our content', fine-tuning on internal documents instead of retrieving them, and then re-fine-tuning on every document update.
Fine-tuning is the more familiar ML workflow, so it feels like the rigorous answer. The failure surfaces later as stale or hallucinated facts, inability to cite sources, and an unmaintainable retraining cadence.
What to do instead
Route by intent: RAG for dynamic, citable, permission-scoped knowledge; fine-tuning (or a smaller tuned model) for style, format, tone, tool-calling behaviour, or cost/latency reduction on a stable task. State the tradeoff explicitly - update cadence, citation needs, data volume, and per-user access control - when answering design questions.
Declares the application 'evaluated' after reading a handful of outputs, or runs a single LLM-judge with the same model that generated the answer and no rubric, no labelled ground truth for retrieval, and no tracking of prompt or model version between runs.
Manual review is fast for ten examples and judges feel objective. Without ground truth and versioned experiments, quality changes are indistinguishable from noise and regressions ship silently.
What to do instead
Build an evaluation set with expected answers and expected source documents; measure retrieval separately (recall/precision of retrieved chunks) from generation (groundedness, correctness, relevance) using rubric-based judges, including a different judge model from the generator. Run it through the evaluation tooling you would use in production, keep the results in the experiment, and re-run the same set for every prompt, model, or retrieval change.
Deploys a working notebook chain to a model serving endpoint without packaging requirements: relies on notebook-scoped %pip installs or local DBFS paths, hardcodes secrets or API keys, and assumes the endpoint identity can read the backing tables, secret scope, or external model credentials.
The notebook environment already has the libraries, mounted paths, and the developer's permissions, so nothing fails in development. Serving runs in a different container with a different identity and no notebook state.
What to do instead
Declare dependencies and versions in the logged model/chain, use relative or governed paths (Unity Catalog volumes rather than local paths), fetch credentials from a secret scope the endpoint's identity is granted, and test in a staging endpoint with the same permissions as production before promoting.
Monitors only endpoint latency and error rate after launch: no inference table capturing requests and responses, no capture or storage of user feedback, and no quality or drift checks on retrieval and answers.
Latency is the easy metric with a ready dashboard, and quality regressions produce successful HTTP 200s. Without logged payloads and feedback there is no data to detect or debug degradation.
What to do instead
Enable inference tables and payload logging on the endpoint, capture user feedback (thumbs, edits, corrections) tied to a request ID, and monitor both operational metrics and quality metrics - groundedness, refusal rate, retrieval hit rate, and input drift - with alerts routed to an owner and a review loop that feeds failures back into the evaluation set.
Assumes Unity Catalog table permissions automatically govern what the RAG application retrieves: the retriever queries all indexed chunks with one service identity, so any user of the app can receive content from documents they are not entitled to see.
Permissions are set on tables and the index looks 'governed', but the application is the only principal touching the data. Candidates conflate data governance of the store with enforcement inside the retrieval path.
What to do instead
Enforce access control at retrieval time: store document-level ACL/group tags in chunk metadata, resolve the end user's identity and groups, and filter (or pre-filter) the vector search query accordingly before the chunks reach the prompt. Test with two users who have different entitlements.
Logs raw prompts, retrieved chunks, and model responses - including customer PII or regulated data - to inference tables and trace stores without redaction, and ships with no input/output guardrails against prompt injection, jailbreaks, or unsafe content.
Logging everything is the default for debugging and tracing is required for evaluation, so sensitive data flows into tables with broader read access than the source. Guardrails are treated as a later hardening step rather than part of the design.
What to do instead
Classify data before it enters the prompt, mask or redact PII in logged payloads, restrict access to inference and trace tables to the minimum group, set retention, and place guardrails at both input and output - including instruction-injection detection and filtering of retrieved content treated as untrusted input. Document the responsible-AI controls alongside the architecture.