Foundation models and prompt engineering
Domain 2 - Foundation models and prompt engineering
This domain tests how well you choose a model and shape a prompt, and whether you can identify which parameter produced a bad output.
Model choice and limits
Every foundation model has a context window; prompt tokens plus max_new_tokens must fit inside it. Chat models expect roles (system, user, assistant) and have a model-specific prompt template - use the SDK chat method so the template is applied for you. Granite, Llama and Mistral do not share identical templates or tokenizers.
Decoding parameters
- Greedy decoding always takes the highest-probability token; deterministic and right for extraction and classification.
- Temperature scales randomness; 0 is near-deterministic, high values increase variety and hallucination risk.
- Top_p / top_k truncate the candidate distribution - not the same control as temperature.
- Repetition penalty suppresses loops; too high produces awkward text.
- Stop sequences end generation early and save tokens.
- Random seed makes sampling reproducible.
Worked example
You need a strict JSON label from 5,000 tickets. Set greedy decoding, temperature 0, max_new_tokens around 20, a stop sequence at the closing brace, and validate the output against a schema. Reasoning: any sampling randomness makes labels non-reproducible, and a large token budget invites commentary around the JSON.
Traps
- Thinking max_new_tokens is the total window rather than the output cap.
- Adding few-shot examples that exceed the window, silently truncating the system prompt.
- Raising temperature to fix a hallucination.
- Trusting free text where a schema check is needed.
- Reusing one prompt across model families without checking the template.