AI-300 common mistakes
AI-300 · Understand the traps and how to avoid them.
Designing every model deployment as a real-time managed online endpoint, even when the described workload is a nightly bulk scoring job over a large table.
Candidates pattern-match 'deploy the model' to 'online endpoint' and skip the step of classifying the workload pattern (batch, real-time request/response, streaming) before choosing a compute target, so they pick an architecture that idles expensive always-on capacity and cannot handle the volume.
What to do instead
Before any infrastructure question, classify the workload: interactive low-latency requests -> online endpoint with autoscaling; scheduled bulk scoring over a dataset -> batch endpoint/job; event-driven -> streaming consumer. Then size compute to the pattern and justify the choice in terms of cost, latency, and throughput.
Embedding storage account keys or service keys directly in job/deployment YAML or environment variables, and granting workspace-wide owner roles to make the pipeline 'just work'.
It is the fastest path to a working sample, so candidates default to it and never revisit least privilege; they also confuse 'the pipeline succeeded' with 'the infrastructure is correct'.
What to do instead
Use workspace datastores backed by credential-less auth (managed identity / workspace connection to Key Vault), grant the narrowest role at the narrowest scope (e.g., blob data reader on the specific container), and rotate/centralize secrets in Key Vault. Expect exam items that ask which identity needs which role on which resource.
Re-registering a model under the same name and overwriting the artifact instead of creating a new immutable version, then trying to roll back a bad deployment.
Candidates treat the model registry like a file share and optimize for 'clean' names; they also conflate model version with deployment name, so they believe a rollback only requires redeploying the previous code.
What to do instead
Register each training run as a new numbered version under a stable model name, keep lineage to the run/commit/data asset, and use a mutable alias or tag (e.g., 'candidate', 'production') to point at a version. Rollback = repoint the alias and redeploy that version.
Configuring a drift monitor with label-based accuracy metrics on production data that has no ground-truth labels arriving in real time, or treating feature-distribution drift as proof the model is wrong.
Candidates memorize 'monitor for drift' without distinguishing data drift (input distribution shift), concept drift (input-output relationship shift), and delayed-label performance degradation, so they wire up metrics that can never be computed.
What to do instead
Pick signals that actually exist at the required latency: input feature distribution drift and prediction distribution for unlabeled data; performance metrics only after labels arrive (with a lag alert on label freshness). Add a concept-drift check after label join, and set drift thresholds plus a retraining trigger explicitly.
Selecting the default pay-as-you-go (standard, token-billed) deployment for a generative AI workload described as high, steady volume with strict latency SLOs, instead of provisioned throughput.
Candidates focus on 'which model' and skip capacity planning; they do not connect sustained token-per-minute demand and latency guarantees to the provisioning model, and they forget that quota is per subscription/region and must be requested.
What to do instead
Estimate peak tokens per minute from prompt+completion sizes and request rate; compare standard (variable latency under load, per-token billing) with provisioned throughput (reserved capacity, predictable latency, higher fixed cost). Also state the quota-increase step and a fallback region/deployment for spillover.
Leaving the orchestration logic (prompt assembly, retrieval, tool calls, output parsing) in application code while only the model deployment is provisioned as infrastructure.
Candidates equate 'GenAIOps infrastructure' with 'deploy a model', so the prompt templates, retrieval pipeline, and evaluation config end up unversioned and untested, making the system unreproducible when a prompt or index changes.
What to do instead
Treat the orchestration as a versioned asset (prompt flow / pipeline definition) checked into source control, deployed through the same pipeline as the model, with each prompt and index version pinned to an evaluation baseline. Deployment must reference explicit asset versions, never 'latest'.
Building a vector index with one embedding model and then querying it with a different (or upgraded) embedding model, or changing chunking without rebuilding the index.
Candidates treat the index as an independent component and assume embeddings are interchangeable because the text 'looks the same'; dimension mismatches or embedding-space mismatches then surface only as vague quality loss, which is hard to attribute under exam pressure.
What to do instead
Pin the embedding model and version (plus chunk size/overlap) as metadata on the index, reject queries whose embedding deployment differs, and require a full re-index plus evaluation run as part of any embedding or chunking change. Verify failure behavior explicitly (dimension mismatch should be a hard error, not silent degradation).
Accepting a single aggregate quality score (e.g., average groundedness or a lone judge score) as proof the generative AI system is acceptable, without per-slice breakdown or judge calibration.
Aggregate metrics are easy to report and hide the tail; candidates also assume an LLM judge is objective and skip validating it against human-labeled samples, so systematic judge bias is invisible.
What to do instead
Report metrics by slice (intent, language, document type, prompt length) and inspect the worst-performing slice, not the mean. Calibrate any LLM judge against a human-annotated set, report agreement, and keep deterministic metrics alongside model-graded ones.
Enabling request/response content logging for observability without defining PII handling, retention, and access controls, or logging only the final answer and not the retrieved context.
Candidates want traces for debugging and turn everything on; they also think the answer alone is enough to diagnose a bad response, so groundedness and retrieval failures become unattributable after the fact.
What to do instead
Log the full trace (query, retrieved chunks with document IDs and scores, prompt version, model deployment, token counts, latency) tied by a correlation ID, and pair it with redaction/PII filtering, retention limits, and role-restricted access. Alert on both quality signals and operational signals (latency, token usage, error rate, throttling).
Recommending fine-tuning as the first optimization for quality problems, or lowering temperature and shortening prompts as the whole cost/latency strategy, without measuring where tokens and latency are actually spent.
Fine-tuning feels like the most 'advanced' answer, and candidates skip the cheaper, higher-leverage levers (prompt and context optimization, retrieval precision, model routing/cascades, caching) because they have not decomposed the failure into retrieval, prompting, or model capability.
What to do instead
Diagnose first: if the answer is ungrounded, fix retrieval/chunking; if format is wrong, fix prompt/examples; if the task is genuinely novel-format or style-heavy, consider fine-tuning or distillation. For cost/latency, measure the token split (system prompt, context, history, output), then apply context trimming, response caching, smaller-model routing with escalation, and streaming, validating each change against the same evaluation harness.