Capabilities and Limitations of Generative AI
Domain 2 tests judgement: can you decide whether generative AI is the right tool for a specific task, and what safeguards that task needs? Questions pair a plausible-sounding claim with a scenario and ask you to spot the limitation.
Where it performs well
- Transforming existing text: summarising, rewriting for a tone or reading level, translating, extracting fields into a schema.
- Classification and routing at volume, where a human or a rule can spot-check.
- First drafts, brainstorming and code assistance for well-known patterns.
- Multimodal tasks such as describing an image or reading a scanned form.
Where it fails predictably
- Hallucination: fluent, confident, fabricated facts, citations or figures. There is no internal truth check.
- Knowledge cut-off: no awareness of recent events or private data unless you provide it.
- Context limits: long inputs get truncated or lose salience in the middle.
- Exact computation: arithmetic, counting, statutory rules and precise citations are unreliable without a calculator or code tool.
- Non-determinism: the same prompt can give different answers; consistency must be engineered and tested.
- Bias: training-data skew can produce discriminatory or stereotyped output.
- Security: models do not reliably separate instructions from data, so untrusted text can hijack behaviour.
A suitability test
Ask four questions: is the task transformation rather than precise computation; is a source of truth available to check against; is a wrong answer cheap to catch before harm; does the process tolerate variation? Two or more 'no' answers means redesign or reject.
Worked example
Two candidate tasks: (a) cluster 200 support tickets into themes; (b) calculate statutory redundancy pay for 40 employees. Choose (a). Reasoning: it is a transformation with a verifiable source - the tickets themselves - errors are visible on review, and a poor theme costs little. Task (b) needs exact statutory arithmetic and legal interpretation; a confident wrong figure creates legal and financial exposure. Use generative AI only to draft the explanatory letter around a figure computed deterministically and checked by a qualified person.
Traps that cost marks
- Equating fluent output with verified output.
- Believing web browsing or a tool call removes hallucination rather than reducing some classes of it.
- Extrapolating from a handful of good demo prompts to production reliability.
- Confusing 'the model can do this' with 'it meets the accuracy this process requires'.