watsonx AI Assistant Engineer cheat sheet

Architecture layers: channel integration → orchestration/runtime (dialog, actions, search) → natural language understanding → knowledge and model services (LLM, vector index) → logging and analytics.

Two build paradigms coexist: classic dialog nodes (tree-based, intent/entity driven) and actions (LLM-assisted, step-based, intent-optional); a single assistant can contain both.

Core object hierarchy: instance (region/plan) → assistant → environments (draft/live) → integrations/channels; knowledge bases and extensions attach at assistant level.

Intents represent a goal with example utterances; entities represent typed parameters with synonyms and patterns; contexts hold runtime variables with scope (turn, session, skill/action, assistant).

Stateless vs stateful: the runtime maintains session state keyed by session ID; stateless API calls require resending context each turn.

Countertop (web chat) and preview are testing surfaces; preview reflects draft changes immediately, while the live environment requires explicit publication.

Webhook/extension calls occur at defined lifecycle points (pre/post message, action step, dialog node); response time budget should stay within channel timeout (typically ~5–8s for web chat).

Plan tiers govern feature availability (e.g., phone/voice, advanced analytics) and API rate limits; region choice affects data residency and available models.

Design from user goals, not from org chart: enumerate top tasks by frequency × business value, then map each to an action or dialog flow.

Action naming rule: action name should read as a user request (verb + object), because the name plus description is used for action disambiguation by the LLM.

Disambiguation pattern: when multiple actions match, the assistant asks a clarifying question listing candidate actions; keep candidates to ≤5 with distinct names.

Slot-filling rule: ask for one required slot at a time, mark optional slots optional, and offer a confirmation step before irreversible actions.

Fallback design: two-stage fallback (search knowledge → escalate/transfer), never a bare "I didn't understand"; capture the failed utterance for review.

Tone and persona: define a persona (name, formality, emoji policy, apology style) and apply it consistently across actions, errors, and knowledge answers.

Conversational repair: always provide a way to cancel, restart, or talk to a human, and echo back captured values before submission.

Content design constraint: keep prompts under ~2 lines, avoid jargon, and phrase yes/no questions so a single word answers them.

Action step types: conversational (collect/confirm), advanced (conditions, variable assignment, expression), extension/webhook call, and end-with-response.

Every action begins with an LLM-generated step unless the first step is explicitly set to a fixed conversational step; set "Ask when" conditions to gate entry.

Variables: action-scoped (local) vs session-scoped (persist across actions); set via expression, and reference with ${variable_name} in text and JSON.

Expressions use Spring Expression Language (SpEL); helpers include input.text, sys.* system entities, and string/date utility methods.

Dialog nodes execute top-down with jumps; digressions let a node interrupt and return; slot handlers replace node-level prompts for entity collection.

Multi-turn actions must handle "change previous answer" and "start over" paths explicitly, otherwise captured slots go stale.

Action JSON can be exported/imported for version control; treat actions as code and diff before publishing.

Error handling: wrap extension calls with a check on return code and provide a fallback conversational step on failure.

Model selection trade-off: smaller/faster models for intent routing and slot extraction, larger models for generative answers and multi-step reasoning.

Prompt anatomy used by actions: system prompt (persona, constraints, output format) + action description + step instruction + conversation history + retrieved grounding.

Output control: request structured output (JSON with named fields) when a downstream step parses the result; specify "if unknown, say you don't know".

Temperature: low (≈0–0.3) for deterministic extraction/routing, higher (≈0.5–0.7) for open conversational phrasing.

Hallucination mitigation: constrain the model to provided context, require citations or explicit refusal, and cap answer length.

History management: include only recent turns (token budget), summarize long conversations, and never inject raw PII into prompts unnecessarily.

Prompt evaluation loop: build a labeled set of utterances/answers, measure routing accuracy and answer groundedness, and re-run after prompt or model changes.

Prompt injection defense: treat user text as data, separate instructions from user content, and validate tool/extension arguments before execution.

A knowledge base is a searchable index built from uploaded documents or crawled URLs; it is separate from dialog content and can be shared across actions.

Supported source types typically include PDF, DOCX, HTML, plain text, CSV, and URL crawl; each source has a sync schedule and last-sync status.

Chunking matters: smaller chunks improve precision, larger chunks preserve context; overlap prevents answer truncation at boundaries.

Search types: keyword (lexical), semantic/vector, and hybrid; hybrid generally performs best for mixed terminology.

Set a confidence/relevance threshold; below it, route to fallback rather than answering from weak matches.

Preprocessing rules (headers/footers removal, table handling, language filters) directly change retrieval quality; re-index after rule changes.

Multilingual knowledge bases require either language-specific sources or a multilingual embedding model; verify with cross-language queries.

Test retrieval directly with representative queries before wiring the knowledge base into an action's answer step.

RAG pipeline: query rewrite/expansion → retrieval (top-k) → rerank/filter → prompt assembly with context → grounded generation → citation/answer.

Top-k trade-off: too few chunks miss the answer, too many dilute the prompt and increase cost/latency; typical k is 3–10 before reranking.

Grounding rule: the model may only assert facts present in retrieved context; otherwise it must say the information is unavailable.

Chunk metadata (source title, URL, section) enables citations and lets users verify answers; surface links in the response.

Answer quality metrics: retrieval recall@k, answer faithfulness/groundedness, answer relevance, and refusal correctness.

Failure modes: stale index, chunk boundary splits, vocabulary mismatch (fix with synonym expansion or hybrid search), and conflicting sources.

Conversational RAG needs query rewriting from dialogue context (e.g., "what about the second one?") before retrieval.

Latency budget: retrieval + rerank + generation; cache frequent queries and stream answers to mask generation time.

Custom extension pattern: OpenAPI 3.0 spec describes the service; watsonx Assistant imports operations as callable steps with typed parameters.

Authentication options for extensions: basic auth, bearer/API key header, and OAuth; secrets belong in the extension configuration, not in dialog text.

Message API endpoint shape: POST /v2/assistants/{assistant_id}/message with session context; response returns output text, context, and user-defined actions.

Session lifecycle: create session, send messages with session_id, delete session at end; stateless calls pass context without a session.

Channel integrations (web chat, Slack, Teams, phone/SMS, custom) each impose their own payload and timeout constraints; normalize in the action layer.

Webhooks can call external systems mid-dialog for lookups (order status, CRM) and return variables consumed by later steps.

Idempotency and retries: external calls should be idempotent or guarded, because channel retries can duplicate side effects.

Log all extension requests/responses (redacted) to debug failures; return user-safe error messages rather than raw HTTP codes.

Key metrics: total conversations, containment/resolution rate, escalation rate, fallback rate, action completion rate, and average turns per resolution.

Unmatched/fallback utterances are the primary backlog: cluster them, then add actions, synonyms, or knowledge content.

Conversation logs support review of full transcripts; filter by action, channel, date, and outcome to find failure patterns.

A/B or controlled rollout: publish changes to a subset or compare draft vs live metrics over comparable periods before full release.

Quality gates: regression test set of utterances with expected action/knowledge answers, run before each publication.

LLM answer evaluation requires human or model-based scoring of groundedness; log retrieved sources alongside answers to audit them.

Track latency and error rates per extension; degradation in a downstream API often appears first as fallback spikes.

Close the loop: each identified gap becomes a tracked change with an owner, then re-measured after release.

Least privilege: API keys and service credentials scoped to the minimum role; rotate keys and never embed them in client-side web chat code.

PII handling: redact or mask sensitive data before sending to LLM or logs; configure log redaction and retention per policy.

Access control: role-based roles (admin, editor, viewer) at instance/assistant level; restrict who can publish to live.

Prompt injection and data exfiltration: validate extension arguments, restrict which tools an action can call, and never let model output trigger arbitrary URLs.

Responsible AI practices: disclose that users are interacting with an AI, provide human escalation, and avoid high-stakes automated decisions without review.

Governance artifacts: model cards/use-case documentation, data provenance for knowledge sources, and a change log for prompt and model updates.

Compliance mapping: data residency by region, encryption in transit and at rest, and audit trails for configuration changes.

Bias and fairness checks on evaluation sets: test names, dialects, and languages, and review refusal patterns for disparate impact.

Environment model: draft (editable, previewable) and live (published); publishing promotes the current draft snapshot to live.

Promotion discipline: test in preview, run regression set, then publish; keep an export of actions/dialog JSON as a rollback point.

Channel setup requires channel-specific credentials (e.g., Slack app tokens, Twilio number, web chat embed script) stored per environment.

Web chat embedding uses a script snippet with integration ID and region URL; version the snippet and restrict allowed domains.

Multi-environment pattern: separate assistants or instances for dev/test/prod, with configuration values injected per environment.

Rollback: republish a prior snapshot or re-import saved JSON; verify channel credentials still match after rollback.

Release checklist: knowledge base re-synced, extensions reachable, secrets valid, analytics tagging correct, and escalation path staffed.

Monitor post-deploy: watch fallback, error, and latency metrics for the first hours after publication to catch regressions.