watsonx GenAI Engineer cheat sheet

watsonx is organized as three product pillars plus a shared studio: watsonx.ai (build/tune/deploy foundation models), watsonx.data (open lakehouse for governed data access), watsonx.governance (risk, compliance, lifecycle oversight), with watsonx Orchestrate for agentic workflows.

watsonx.ai runtime services include the prompt lab, foundation model library, tuning studio, deployment spaces, and the inference API; the control plane (provisioning, IAM, cataloging) is separate from the inference data plane.

Resource hierarchy: IBM Cloud account > resource group > service instance > project or deployment space. A project is a collaborative authoring container; a deployment space is the runtime container that hosts deployed assets and is where inference endpoints live.

A model asset becomes an inference endpoint only after promotion from a project to a deployment space and creation of an online deployment; the deployment returns a scoring/chat endpoint URL plus an IAM bearer token requirement.

Identity and access: IBM Cloud IAM API keys authenticate service calls; project and space collaborators get roles (Admin, Editor, Viewer) and fine-grained access can be scoped to assets such as connections and models.

watsonx.data is built on an open lakehouse architecture (Presto/Spark engines over object storage with Iceberg-style open table formats) and exposes data to watsonx.ai through connections rather than copies.

Prompt Lab supports both chat and freeform modes; chat mode uses role-tagged messages (system, user, assistant) and is the correct mode for instruct-tuned chat models, while freeform mode sends a single composed prompt.

Token accounting drives cost and limits: input tokens plus generated tokens must fit the model's context window, and max_new_tokens caps only the output portion.

The SDK path is the Python package ibm-watsonx-ai (APIClient, ModelInference, FoundationModels, PromptTemplateManager), which wraps the same REST endpoints used by the UI.

Decoding parameters: temperature scales randomness (0 = near-greedy), top_p is nucleus sampling over cumulative probability, top_k restricts to the k most likely tokens; tune one of temperature/top_p at a time rather than all three.

max_new_tokens bounds output length only; stop_sequences halt generation at a literal string; repetition_penalty (>1) discourages loops but too high a value degrades fluency.

Zero-shot prompting supplies only instructions; few-shot prompting adds labeled exemplars in the prompt; both differ from fine-tuning, which changes weights.

Chain-of-thought prompting asks for intermediate reasoning steps and improves multi-step arithmetic/logic tasks, at the cost of more output tokens.

Instruction clarity rule: specify role, task, input data, output format, and constraints explicitly; ambiguous pronouns and missing format specifications are the most common cause of unusable output.

Structured output is obtained by describing the exact schema (for example JSON keys and types) in the prompt and optionally providing one exemplar; constrained decoding or post-validation is still required for strict schema guarantees.

Prompt templates separate fixed instructions from variable slots; watsonx.ai PromptTemplateManager stores templates as project assets so the same template can be versioned and reused across deployments.

Model selection criteria: context window size, latency, cost per token, language coverage, license/indemnification terms, and whether the task needs chat-style instruction following versus base-model completion.

A system prompt sets persistent behavior and guardrails for a chat model; user turns carry the task, and prior assistant turns are included as history, consuming context budget on every call.

RAG pipeline stages: ingest and parse documents, chunk, embed chunks, store vectors with metadata, embed the query, retrieve top-k by similarity, optionally rerank, then construct a grounded prompt with the retrieved passages and citations.

Chunking trade-off: chunks must be small enough that several fit in the context window yet large enough to be self-contained; overlap (commonly 10-20 percent) preserves facts that straddle boundaries.

Similarity metrics: cosine similarity compares vector direction and is standard for normalized text embeddings; dot product equals cosine when vectors are unit-normalized; Euclidean distance is order-equivalent for normalized vectors.

An embedding model and the query/document encoder must match; mixing embeddings from two different models in one index produces meaningless similarity scores.

Vector index choices in watsonx.data and vector stores: exact flat search is accurate but slow at scale; approximate nearest neighbor indexes (HNSW, IVF) trade recall for latency and require tuning of ef_search or nprobe.

Metadata filtering applies structured predicates (source, date, ACL) before or alongside vector search and is the primary mechanism for permission-aware retrieval.

Reranking with a cross-encoder scores query-passage pairs jointly and typically improves precision over bi-encoder retrieval, at added latency, so it is applied to a small candidate set (for example top 50 to top 5).

Grounding and faithfulness: instruct the model to answer only from provided context and to say it does not know otherwise; attach chunk identifiers so answers can cite sources.

Common RAG failure modes: retrieval miss (answer not in top-k), context dilution from too many chunks, chunk truncation, stale index after document updates, and prompt injection carried inside retrieved text.

Customization ladder, cheapest to most expensive: prompt engineering, RAG, then parameter-efficient tuning (LoRA/adapters), then full fine-tuning, then pre-training from scratch.

LoRA freezes base weights and trains low-rank matrices A and B with rank r; trainable parameters scale roughly with r times the sum of adapted layer dimensions, so r=8 or 16 is a typical starting point.

Tuning in watsonx.ai is configured with a foundation model, a JSONL/JSON training dataset, a task type (classification, extraction, generation, summarization, question answering), and compute/epoch settings.

Training data format matters: instruction tuning expects prompt-completion or question-answer pairs; classification tuning expects text plus label fields; malformed or unbalanced records are the most frequent cause of failed tuning jobs.

Hyperparameters: learning rate, number of epochs, batch size, and LoRA rank/dropout; too many epochs on small data causes memorization and degraded generalization.

Evaluation of a tuned model compares it against the base model on a held-out set using task metrics (accuracy/F1 for classification, ROUGE/BLEU or human/LLM-judge ratings for generation).

Tuning versus RAG decision rule: use RAG when knowledge changes frequently or must be cited; use tuning when the goal is style, format, domain vocabulary, or task behavior that prompts cannot reliably enforce.

A tuned model is stored as a model asset in the project, then promoted to a deployment space and deployed like any foundation model, producing a new inference endpoint.

Guardrail: keep a held-out validation split and check for data leakage (evaluation examples appearing in training data) before declaring improvement.

watsonx.governance manages the AI lifecycle across build, evaluate, risk assessment, approval, deployment, and monitoring, and it can govern models built outside watsonx (for example third-party or open-source models).

Core governance objects: AI use case (fact sheet capturing purpose, owners, and risk), model inventory entry, model evaluation report, risk assessment, and approval workflow.

Model evaluation dimensions include quality (accuracy, F1), fairness (disparate impact, demographic parity difference), drift (changes in input or output distributions over time), and generative-specific metrics such as faithfulness, answer relevance, and toxicity.

Fairness metrics compare outcome rates across protected groups; disparate impact ratio near 1 indicates parity, and common regulatory thresholds flag ratios below 0.8 (the four-fifths rule).

Drift monitoring compares production data distributions against a baseline using statistical tests; drift detection triggers review rather than automatic retraining.

Regulatory mapping: the EU AI Act defines risk tiers (unacceptable, high, limited, minimal) with documentation, human oversight, and logging obligations for high-risk systems; NIST AI RMF organizes work into Govern, Map, Measure, Manage functions.

Governance requires lineage and traceability: prompts, model versions, training data references, evaluation results, and approvals should be recorded so an auditor can reconstruct how a decision was produced.

Human-in-the-loop controls and fact sheets document intended use, limitations, and known risks; the fact sheet is the artifact regulators and internal reviewers most often request.

Separation of duties: the person who approves a model for production should not be the sole author of its risk assessment.

LangChain building blocks used with watsonx: model wrappers (chat and LLM), prompt templates, output parsers, retrievers, document loaders, and chains that compose them.

A typical RAG chain is: retriever -> prompt template that injects context -> chat model -> output parser; LCEL pipe syntax composes these steps and enables streaming and async execution.

watsonx Orchestrate exposes skills/tools that agents invoke; agent design requires explicit tool descriptions, input schemas, and a fallback path when no tool matches.

Function/tool calling pattern: declare tools with JSON schemas, let the model emit a tool-call request, execute it in application code, then return the result as a new message so the model can continue.

Conversation memory must be bounded: keep a sliding window or summarize older turns, because full history grows linearly and eventually exceeds the context window.

Guardrails belong at both input and output: validate and sanitize user input, filter retrieved content for prompt injection, and post-check model output for policy violations and schema conformance.

Streaming responses improve perceived latency but complicate post-processing; buffer or incrementally validate when output must conform to a schema.

Error handling: inference calls can fail with 429 (rate limit) or 5xx; implement retry with exponential backoff and jitter, plus timeouts and idempotent fallbacks.

Secrets such as IBM Cloud API keys must be injected from environment variables or a secrets manager, never hard-coded in notebooks or application source.

Deployment types in watsonx.ai: online deployment for real-time synchronous scoring, and batch deployment for scheduled scoring over large datasets; both live in a deployment space.

Online deployment sizing is chosen at creation (small, medium, large hardware specs); changing size requires redeploying, so capacity planning precedes go-live.

Deployment lifecycle states include deploying/preparing, deployed, updating, and failed; a deployment must reach the deployed state before its endpoint returns successful responses.

Monitoring signals for generative apps: request volume, latency percentiles, error rates (401 auth, 429 throttling, 5xx), token consumption, and cost per request.

Quality monitoring in production uses sampled evaluation (faithfulness, answer relevance, toxicity) plus drift detection on inputs and outputs; alerts fire when metrics breach configured thresholds.

Versioning: model assets, prompts, and deployments should be versioned together so a rollback restores a known-good combination rather than just the model.

Rollback strategy: keep the previous deployment available until the new one passes smoke tests, then shift traffic; deleting a deployment invalidates its endpoint URL.

Logging must balance observability and privacy: redact PII before persisting prompts and completions, and set retention consistent with policy.

Access to inference endpoints requires a valid IAM bearer token; expired or mis-scoped API keys surface as 401/403 rather than model errors.

Document ingestion pipeline: connect to source (object storage, databases, SaaS), extract text, clean (strip boilerplate, headers/footers, encoding artifacts), normalize, chunk, embed, and index with metadata.

watsonx.data connections define how watsonx.ai and other services reach data sources; credentials are stored in the connection asset, not in application code.

Document loaders must handle format diversity: PDF text extraction can lose tables and multi-column layout, so OCR or layout-aware parsers are needed for scanned or complex documents.

Metadata design drives retrieval quality: capture source, section, page, timestamp, author, and access-control tags at ingest time, because retrofitting metadata later requires reprocessing.

Deduplication before embedding reduces index bloat and repeated citations; near-duplicate detection can use hashing or embedding similarity thresholds.

PII handling: detect and redact or tokenize sensitive fields during preparation; regulations constrain storing raw personal data in vector indexes just as in any other store.

Refresh strategy: incremental re-indexing on document change keeps retrieval current; full rebuilds are needed when the embedding model or chunking scheme changes, since old vectors become incompatible.

Data quality checks: verify chunk length distribution, embedding count versus document count, null metadata rates, and sample retrieval queries before connecting the index to a production application.

Governed data access via watsonx.data means retrieval honors the same access policies as the underlying tables, supporting permission-aware RAG.