Natural Language Processing and Generative AI
Core concepts
NLP turns text into representations machines can use. Know tokenization (splitting text into tokens), embeddings (dense vectors where similar meanings are near), and the transformer architecture built on self-attention, which weighs relationships between all tokens. Understand the pretrain-then-fine-tune pattern and the distinction between encoder models (understanding tasks like classification) and decoder models (generation).
Generative AI
Generative systems produce new content-text, images, code. For language models, know that they predict the next token probabilistically, which explains fluent but sometimes false output. Key terms: prompt, context window (maximum tokens processed at once), temperature (higher = more random), hallucination (confident fabrication), and RAG (retrieval-augmented generation, grounding answers in retrieved documents). Understand fine-tuning versus prompting: the first changes weights, the second does not.
Traps
Questions often imply models "look up" facts; they generate from learned patterns, so citations can be invented. Do not confuse token with word-tokens are subword units, so counts differ. Beware claims that a larger context window guarantees accuracy; long contexts can dilute attention. Also separate training data cutoff from real-time knowledge-models lack live access unless given tools.
Rule of thumb
For factual, source-grounded answers, prefer retrieval-augmented generation over trusting parametric memory; retrieval supplies evidence the model can cite.
Worked example
A chatbot invents a plausible court case. The fix is not a bigger model but RAG: retrieve verified documents, instruct the model to answer only from them, and require citations-reducing hallucination and making errors auditable.