Generative AI Applications, RAG, and Agents
This domain tests retrieval-augmented generation design choices and how agents add planning and tool use on top of a foundation model.
The RAG pipeline
Ingest → parse (PDF, DOCX, HTML) → chunk with size and overlap → embed with an embedding model → index in a vector store (Milvus in watsonx.data, OpenSearch, FAISS) → retrieve top-k, optionally metadata-filtered and re-ranked → compose a prompt with the retrieved context → generate with citation and refusal instructions.
Design choices
Chunk size trades precision against context; overlap preserves sentences at boundaries; store source, page and date metadata for filtering; use the same embedding model at index and query time; top-k too small misses evidence, too large dilutes the prompt.
Evaluation
Retrieval: context precision and recall. Generation: faithfulness or groundedness, answer relevance, fluency. High-stakes domains still need human review.
Agents
An agent adds planning and tool calls: the model selects a tool, observes the result, and iterates in a ReAct-style loop. Tools must be permissioned, loop-bounded and idempotent where possible.
Worked example
A policy assistant answers from 400 PDFs and hallucinates. Reasoning: chunk at roughly 500–800 tokens with 10–15% overlap, embed and index, retrieve the top 5 with source metadata, instruct "answer only from context, otherwise say not found", then measure faithfulness on a labeled question set. Raising temperature would make it worse.
Traps
Stuffing whole documents into the prompt; embedding-model mismatch between indexing and querying; retrieving context but forgetting to include it; unbounded agent loops; no refusal path.