AIAI common mistakes
AIAI · Understand the traps and how to avoid them.
Treating 'AI' as a single technology and accepting answer options that describe any automated or rule-based system as AI, rather than checking whether the option describes a system that learns patterns from data.
Everyday usage of 'AI' is loose, and many distractor options are plausible-sounding automation (scripted workflows, deterministic rules engines) that a hurried candidate labels as AI.
What to do instead
Before choosing, classify each option: does it derive behaviour from data/learned parameters (ML), or is it fixed logic? Eliminate options that are pure rule-based automation when the stem asks about AI/ML capability.
Confusing the roles of training, validation, and test data, e.g. selecting 'use the test set to tune hyperparameters' or 'evaluate final performance on the training set' as correct.
The three splits are taught together and the terms sound interchangeable; exam stems often describe a workflow step and expect you to map it to the correct split.
What to do instead
Memorise the invariant: training fits parameters, validation selects/tunes, test is touched once for an unbiased final estimate. Reject any option that leaks the test set into development decisions.
Mixing up overfitting and underfitting, or choosing 'add more features / train longer' as the remedy when the described symptom is a large gap between training and validation performance.
Both are framed as 'the model performs badly', and candidates key on the bad performance rather than on which dataset the bad performance appears in.
What to do instead
Anchor on the diagnostic signature: low training error + high validation error = overfitting (regularise, simplify, more data, early stopping); high error on both = underfitting (more capacity, better features, longer training).
Answering data-quality questions by picking 'remove all rows with missing values' or 'drop outliers' as a default, without checking whether the missingness is informative or the outliers are valid signal.
Dropping is the simplest operation and feels safe, but exam items usually set up a scenario where deletion introduces bias or loses a class.
What to do instead
First classify the mechanism (MCAR/MAR/MNAR) and the proportion affected, then choose imputation, flagging, or deletion accordingly; treat outliers as data errors only when there is evidence they are errors.
Treating data leakage as a modelling problem rather than a pipeline problem, e.g. accepting 'the model is too complex' as the explanation for suspiciously perfect validation scores.
Candidates associate high accuracy with good models and do not consider that preprocessing (scaling, target encoding, imputation) was fit before the train/test split or across the full dataset.
What to do instead
When scores look too good, suspect leakage first: check that all transforms are fit on training folds only and that no target-derived or future information enters features.
Explaining hallucinations as the model 'lying' or 'not knowing it should tell the truth', and choosing options about model intent or honesty rather than about next-token prediction without grounding.
Anthropomorphising the model is intuitive, and distractor options are written in intentional language that sounds explanatory.
What to do instead
Describe LLM behaviour mechanically: probabilistic token generation, no built-in fact database, no access to ground truth unless retrieved or provided in context. Prefer fixes involving grounding/RAG, citations, and constrained outputs.
Confusing fine-tuning, RAG, and prompt engineering, e.g. recommending fine-tuning to inject frequently changing factual knowledge, or RAG to change output style/format permanently.
All three are 'ways to improve an LLM' and the distinctions blur under time pressure.
What to do instead
Use the decision rule: changing knowledge that updates often -> retrieval (RAG); changing behaviour/style/task format persistently -> fine-tuning; quick behavioural adjustment without retraining -> prompting. Match the option to the nature of the change.
Treating fairness as satisfied once a single statistical parity metric is met, and rejecting options that describe trade-offs, context-dependence, or the need to define the protected group and harm first.
Candidates look for one 'correct' fairness formula, but fairness metrics are mutually incompatible in general and the exam expects recognition of that.
What to do instead
Frame fairness as a context-dependent choice among competing metrics with explicit trade-offs against accuracy; prefer answers that specify who is affected, which harms matter, and that acknowledge you cannot maximise all parity metrics simultaneously.
Conflating privacy, security, and governance controls, e.g. answering a privacy question with 'encryption at rest' or a governance question with 'penetration testing', instead of the control that addresses the stated risk.
The domains overlap heavily in vocabulary (PII, access control, audit), and candidates pattern-match on keywords rather than on the risk being mitigated.
What to do instead
Name the risk in the stem first (data exposure in prompts, model inversion, unauthorised use, unaccountable decisions), then select the control type that directly mitigates it: minimisation/anonymisation for privacy, adversarial testing/access control for security, policy/audit/accountability for governance.
Choosing the most technically sophisticated option for a business-use-case question, or recommending building a custom model, without checking feasibility, data availability, cost, and measurable business value.
Technical candidates equate sophistication with correctness, while these items reward matching solution complexity to problem constraints and ROI.
What to do instead
Evaluate options against the stated constraints (data volume, latency, budget, regulatory limits, existing tooling) and prefer the simplest approach that meets the success metric; treat 'build from scratch' as a last resort unless the scenario rules out simpler options.