Model Selection, Inference, and Customization
Domain 2 - Model Selection, Inference, and Customization
This domain tests task-to-model fit, inference parameter control, and the three OCI fine-tuning paths. Questions typically describe a workload (summarization, RAG answer synthesis, classification, semantic search) and ask for the model family, parameter settings, or customization method - including the infrastructure consequence.
Core rules
- Match capability to task: Command R/R+ for grounded generation and tool use; Llama and Mistral for open-weight general generation; Embed for vectors; Rerank for reordering retrieved passages. Embedding models do not generate text.
- Inference parameters: max_tokens caps output; temperature, top_p, top_k control sampling; frequency and presence penalties discourage repetition; stop sequences end generation.
- Customization: Vanilla fine-tuning updates all weights (most data and compute); LoRA trains low-rank adapters; T-Few is parameter-efficient and designed for small labeled sets. All require a dedicated AI cluster, and training data is JSONL prompt/completion records in Object Storage.
- Validate on a held-out split; overfitting shows as validation quality falling while training loss drops.
Worked example
A team has 300 labeled examples to make a model answer in a strict JSON schema, and latency must stay low. Reasoning: schema compliance is first a prompting problem - specify the schema and add few-shot examples. If that fails, T-Few fits: 300 examples is small, and T-Few is built for few-shot adaptation at lower cost than Vanilla. Reserve LoRA or Vanilla for larger corpora or deeper behavior change. A dedicated fine-tuning cluster is required, then a hosting cluster for the custom model endpoint.
Traps
Selecting an embedding model for chat; assuming fine-tuning expands the context window; ignoring that a custom model needs hosting capacity after training; confusing LoRA with T-Few; setting both temperature and top_p aggressively.