OCI GenAI Pro cheat sheet

Token: the subword unit a model reads or emits; billing and context limits in OCI Generative AI are measured in tokens, not characters or words.

Temperature controls sampling randomness; top_p (nucleus) and top_k truncate the candidate distribution. Setting temperature near 0 makes output near-deterministic but does not guarantee identical text across model versions.

max_tokens caps generated output length only; the prompt plus output must fit inside the model's context window, so long prompts reduce available output budget.

Embeddings map text to dense vectors; similarity is computed with cosine or dot product. Cosine ignores magnitude, dot product does not, so normalize vectors before using dot product as a cosine substitute.

OCI Generative AI is a managed service exposing pretrained models through an API and console; it offers chat, text generation, and embedding endpoints plus a dedicated inference endpoint for hosting fine-tuned or imported models.

On-demand (shared) inference bills per token and is capacity-limited by shared pool availability; dedicated AI cluster hosting reserves compute for consistent throughput and is the required path for fine-tuned model hosting.

The playground in the OCI Console lets you compare models and tune parameters before writing code; the same parameters map to SDK and CLI calls.

Hallucination is fluent but unsupported output; grounding via retrieval or fine-tuning reduces it but does not eliminate it, so output validation belongs in the application layer.

Prompt design levers: role/system framing, few-shot examples, explicit output format instructions, and delimiters separating instructions from user-supplied content.

Model families on OCI Generative AI include Cohere Command and Embed, Meta Llama, and Mistral, each offered in different sizes; larger variants raise quality and cost/latency together.

Selection rule: choose the smallest model that passes an evaluation set on your task; escalate size only when quality gates fail, because latency and per-token cost scale with size.

Fine-tuning (T-Few style parameter-efficient tuning) adapts a base model to a domain or style using labeled examples; it teaches behavior and format, not new factual knowledge that changes frequently.

Fine-tuned models are hosted on a dedicated AI cluster; you cannot serve a custom model on on-demand shared capacity, so factor cluster cost into the business case.

Retrieval augmentation vs fine-tuning: use RAG when facts change often or must be cited; use fine-tuning when the gap is tone, format, or task-specific reasoning patterns.

Dedicated AI cluster sizing depends on model size and expected concurrency; a cluster hosts one or more endpoints, and scaling is done by adding clusters or endpoints, not by per-request elasticity.

Streaming responses return tokens incrementally, improving perceived latency; total generation time is unchanged, so it is a UX lever, not a throughput lever.

Guardrails and content moderation can be applied to model output; they filter responses but do not fix an under-specified prompt, so combine with prompt design.

RAG pipeline stages: ingest, chunk, embed, store, retrieve top-k, optionally rerank, then compose a grounded prompt with citations.

Chunking rule: chunks must fit the embedding model's input limit and stay semantically coherent; overlap between chunks preserves context across boundaries at the cost of index size.

OCI Search with OpenSearch supports vector fields and k-NN similarity search, making it the managed vector store option inside OCI for RAG.

Similarity metric must match the embedding model's training objective; mixing metrics (for example cosine-trained embeddings queried with unnormalized dot product) degrades recall.

Hybrid retrieval combines keyword/BM25 with vector similarity; keyword handles exact identifiers and rare terms that dense vectors blur.

Reranking a larger candidate set (for example top 50 down to top 5) improves precision but adds latency; it is the standard trade-off knob in production RAG.

Grounding prompt pattern: place retrieved passages in a delimited context block, instruct the model to answer only from that block, and require an explicit fallback when context is insufficient.

Recall failures dominate RAG quality problems: if the answer is absent from retrieved chunks, no prompt engineering recovers it. Measure retrieval recall separately from generation quality.

Metadata filters (tenant, document date, access level) applied at query time enforce both relevance and authorization, preventing cross-tenant leakage through retrieval.

OCI IAM policies gate Generative AI access; least privilege means scoping policies to specific compartments and resource types rather than tenancy-wide generative-ai-family access.

Principals: user principals (console/SDK with user credentials) and instance principals (compute instances) both need explicit policy statements to call the inference endpoints.

Data handling: prompts and completions transit the service; do not place secrets or regulated data in prompts unless your compliance review permits it, and prefer private endpoints for sensitive workloads.

Vault is the OCI service for storing API keys and secrets used by applications calling Generative AI; never embed credentials in code or prompt templates.

Audit captures control-plane API calls; logging and monitoring capture application-level request/response telemetry. Governance requires both, plus retention policy.

Architecture pattern: client to API gateway or load balancer, to application tier with prompt orchestration, to OCI Generative AI endpoint, with retrieval against OpenSearch and secrets from Vault.

Compartment design isolates environments and cost centers; tagging enables cost allocation and policy scoping across generative AI resources.

Prompt injection defense: treat retrieved and user content as untrusted, separate instructions from data with delimiters, and validate or restrict tool/action execution downstream.

Resilience: regional endpoint availability and cluster capacity determine failover options; design a degradation path (smaller model or cached answer) for capacity or quota exhaustion.