MAI common mistakes
MAI · Understand the traps and how to avoid them.
Treating precision, recall, and F1 as interchangeable and choosing the metric with the highest value on an imbalanced test set, rather than selecting the metric that encodes the business cost of each error type.
Candidates memorise the formulas and can compute all three, so they default to whichever number looks best. They forget that on a 1% positive-class problem a model predicting all negatives scores 99% accuracy and high precision is often trivially achievable by being conservative, so the metric choice is a decision about false-positive versus false-negative cost, not a leaderboard.
What to do instead
Before computing anything, state the cost of a false positive and a false negative in business terms (e.g. missed fraud vs. blocked legitimate transaction), then pick the metric that reflects that ratio. Practice explaining why recall is preferred for screening tasks and precision for alerting tasks, and always report the confusion matrix alongside any single scalar so the trade-off is visible.
Confusing L1 and L2 regularisation, or confusing ridge/lasso with dropout and early stopping, when asked what a specific technique does to the loss function or the weight distribution.
All of these are described in study material as 'ways to prevent overfitting', so candidates collapse them into one bucket and answer with the general purpose instead of the mechanism. The exam often asks about the specific effect (sparsity vs. shrinkage, added penalty term vs. modified training loop), and a generic answer scores nothing.
What to do instead
Build a comparison table with one row per technique and columns for: what is added to the objective or training loop, effect on weights, effect on bias/variance, and whether it changes the architecture. Recite the L1 gradient behaviour (constant push toward zero, producing exact zeros) versus L2 (proportional shrinkage) until you can state it without notes.
Reporting a single train/test split result as the model's expected performance, without cross-validation, without a held-out final test set, and without checking for leakage between preprocessing and evaluation.
Candidates learn the mechanics of train_test_split and stop there. They fit scalers, encoders, or target-based aggregates on the full dataset before splitting, which leaks test information, and they tune hyperparameters on the same split they report, producing optimistic numbers they cannot explain when the question asks about generalisation.
What to do instead
Adopt a fixed evaluation protocol you can describe in one sentence: split first, fit all preprocessing inside a pipeline on training folds only, use k-fold cross-validation for model selection, and touch the held-out test set once at the end. When reviewing a scenario, actively look for the phrase 'features were computed on the full dataset' and flag it as leakage.
Describing backpropagation as 'the network learning from its mistakes' or 'the algorithm that finds the best weights' instead of explaining the chain rule computation of gradients and the separate role of the optimiser in applying them.
High-level explanations in introductory material blur the distinction between gradient computation and parameter update, so candidates give a purpose-level answer. Questions that probe understanding of vanishing gradients, learning rate effects, or why an activation function's derivative matters require the mechanism, not the intent.
What to do instead
Write out the forward pass, loss, backward pass, and update step as four distinct stages and be able to say which stage each concept belongs to (e.g. ReLU derivative belongs to the backward pass, learning rate belongs to the update). Practice tracing gradients through two layers by hand once; the mechanism then sticks.
Treating temperature, top-p, and top-k as interchangeable 'creativity' dials, or claiming temperature changes what the model knows, rather than describing them as decoding-time sampling controls over the same next-token distribution.
Product documentation and marketing language describe all three as controls for randomness, so candidates learn the effect but not the layer at which it operates. They then answer questions about determinism, reproducibility, or hallucination as if temperature could add knowledge or guarantee factual accuracy.
What to do instead
Anchor on one sentence: the model produces a probability distribution over the next token; temperature rescales the logits before softmax, top-k truncates to the k most likely tokens, top-p truncates to the smallest set whose cumulative probability exceeds p. Then reason from that: temperature zero approximates greedy decoding, none of these change the training data or guarantee truthfulness.
Answering retrieval-augmented generation questions by saying 'the model is fine-tuned on your documents' or 'the vector database trains the model', conflating retrieval with training.
Both approaches are presented as ways to make an LLM answer questions about private data, so candidates merge them. This produces wrong answers on questions about data freshness, access control, cost of updates, and why a RAG system can still hallucinate when retrieval returns irrelevant chunks.
What to do instead
Draw the two pipelines side by side: fine-tuning changes weights via gradient updates and is stale the moment data changes; RAG keeps weights frozen and injects retrieved context at inference, so updates are index refreshes and permissions can be enforced at retrieval. Memorise the failure modes unique to each (catastrophic forgetting and retraining cost vs. retrieval quality and context-window limits).
Treating fairness as a single property to be maximised, and asserting that removing a protected attribute from the feature set makes a model fair, rather than recognising that multiple fairness definitions are mutually incompatible and that proxies can reconstruct the attribute.
Candidates learn one definition (often demographic parity) and apply it universally. They also accept the intuition that dropping race or gender from inputs removes bias, which ignores correlated features such as postcode or name, and they miss questions asking them to identify which definition is appropriate for a given context.
What to do instead
Memorise at least three definitions (demographic parity, equalised odds, individual fairness) and be able to state a scenario where satisfying one violates another. Treat 'fairness through unawareness' as a named fallacy: removing the attribute does not remove the signal, and proxy variables plus historical labels can reproduce the disparity.
Proposing a deployment architecture without addressing rollback, traffic shifting, or the difference between batch, real-time, and streaming serving, and without naming the monitoring signals that would trigger intervention.
Candidates focus on getting the model to work and treat deployment as an afterthought, so they describe 'deploy to the cloud' as if it were a design. Questions in this domain test whether you can reason about latency budgets, cost per inference, canary releases, and what happens when the model or its input distribution changes.
What to do instead
For every design question, produce four artefacts: the serving pattern and why, the rollout strategy (shadow or canary with a defined metric gate), the rollback trigger, and the top three monitoring signals (input drift, prediction distribution, business KPI). Rehearse one end-to-end answer so the structure becomes automatic.
Monitoring only infrastructure metrics such as uptime, latency, and error rate, and declaring the system healthy while ignoring data drift, concept drift, and degradation of the business outcome the model was built to improve.
Traditional software operations training emphasises availability, and model endpoints do emit those signals, so candidates stop there. They miss that a model can serve 200 OK responses indefinitely while its inputs drift away from the training distribution and its predictions quietly become useless.
What to do instead
Separate monitoring into four layers and name a metric for each: infrastructure (latency, error rate), data (feature distribution drift, schema violations, null rates), model (prediction distribution, calibration, delayed-label accuracy), and business (conversion, fraud loss, escalation rate). When asked about monitoring, list all four and state the retraining or alerting threshold for each.
Framing an AI initiative purely as a technology choice, skipping the problem definition, baseline, success metric, and cost-of-error analysis, and never stating who owns the decision or what the non-AI alternative is.
Technical candidates gravitate to model and tooling questions because those are where they feel competent, so business-framing questions get answered with architecture. This loses the marks attached to problem scoping, stakeholder alignment, and return on investment, and it produces proposals that cannot be evaluated.
What to do instead
Use a fixed opening for any business scenario: the decision being improved, the current baseline and its cost, the metric that will move, the acceptable error profile, and the simplest non-AI alternative. Only after that justify why ML is needed. Practice writing a one-paragraph business case with a quantified baseline before touching the technical design.