Databricks GenAI Engineer study plan

Ingest a document corpus (PDFs/markdown/HTML) into a Bronze→Silver Delta pipeline and strip boilerplate, headers, and navigation text

Split the same corpus three ways (fixed-token, recursive-structure, semantic) and record chunk count, mean chunk length, and overlap for each

Generate embeddings with a Databricks Foundation Model embedding endpoint and persist vectors plus chunk metadata to a Delta table

Create a Vector Search index over the embedding table using Delta Sync and confirm it refreshes on new writes

Add metadata columns (source, section, last_updated, access_level) to the index and demonstrate filtered retrieval queries

Hand-label 50 questions with their correct source chunks and compute recall@5 and recall@10 for the baseline retriever

Build a minimal retrieve→prompt→Foundation Model API chain and capture each step with MLflow tracing

Test one hybrid or re-ranked retrieval variant and record the recall delta against baseline

Author a 50-item evaluation dataset with question, reference answer, and ground-truth context

Implement an LLM-as-judge scorer with an explicit rubric and few-shot calibration examples, then check judge agreement against 15 human-scored items

Score the RAG chain on correctness, groundedness, and context sufficiency using MLflow evaluation

Tune the system prompt and compare two model endpoints or parameter settings against the same eval set

Add a query-rewriting step for follow-up and ambiguous questions and measure its effect on the eval set

Add an input/output validation layer that rejects off-topic or unsafe responses before returning them

Manually review 20 failing eval cases and group them into an error taxonomy (retrieval miss, stale data, hallucination, refusal, formatting)

Refactor the chain into a parameterized, test-covered module callable from notebooks and serving code

Build a chat front end as a Databricks App that calls the chain and streams tokens to the user

Externalize model name, index name, prompt template, and top-k into config stored in Unity Catalog rather than hardcoded values

Add source citations with chunk links to every response and show a retrieval score beside each citation

Handle empty-retrieval and timeout cases with a graceful fallback message and verify via three simulated failure tests

Package the chain as an MLflow logged model and deploy it to a Model Serving endpoint with a defined concurrency and scale-to-zero policy

Enable inference tables on the endpoint and query logged requests and responses to confirm payload capture

Configure Lakehouse Monitoring on the inference table for latency, token count, refusal rate, and retrieval-score drift

Create alerting on error rate and p95 latency thresholds, and run a load test at expected peak QPS

Route 10% of traffic to a second endpoint variant and compare quality and latency against the primary

Apply Unity Catalog catalog/schema/table grants and one column mask or row filter so retrieval cannot surface data a user is not entitled to

Detect and redact PII in source documents before embedding, and re-index the corpus to confirm no PII remains in the vector store

Enable AI Gateway guardrails on the serving endpoint for PII, safety filtering, and rate limits, then verify each blocks an adversarial request

Write a prompt-injection and jailbreak test suite of 15 cases and record pass/fail for each

Draft a model card documenting intended use, data lineage, evaluation results, and known limitations

Take a full-length timed mock exam across all five domains and log every missed item by domain

Re-study the two weakest domains using the missed-item log and re-test those items until correct

Validate end-to-end access control by querying the app as a low-privilege user and confirming restricted content is excluded

Finalize an operations runbook covering rollback to a previous model version, endpoint scaling changes, and re-indexing procedure