Large Language Models and Transformers
Large Language Models and Transformers tests the architecture, pretraining objectives, adaptation methods, and decoding controls that govern modern systems.
Architecture
- Self-attention:
Attention(Q,K,V) = softmax(QKᵀ/√d_k)V. The √d_k scaling keeps large dot products from saturating the softmax. - Multi-head attention runs h projections in parallel, then concatenates and projects.
- Positional information comes from sinusoidal encodings, learned embeddings, RoPE (rotation-based, relative), or ALiBi (distance penalty).
- Decoder-only models apply a causal mask; encoder-only models use bidirectional masked language modelling; encoder-decoder models use span corruption.
Adaptation and decoding
Full fine-tuning, LoRA adapters, instruction tuning, and preference optimization (RLHF, DPO). Decoding options include greedy, beam search, temperature scaling, top-k, and nucleus (top-p) sampling.
Worked example
LoRA on a 4096×4096 projection with rank r = 8 trains matrices A (4096×8) and B (8×4096): 65,536 parameters, against roughly 16.8M for the full matrix - about 0.4%. Reasoning: the weight update is assumed low-rank, so capacity is traded for memory while the frozen base model retains pretraining knowledge and supports many swappable adapters.
Traps
- Describing GPT-style models as bidirectional encoders; the causal mask forbids attending to future tokens.
- Extrapolating far beyond the trained context window without RoPE scaling or ALiBi.
- Expecting temperature 0 to guarantee identical outputs across batched hardware.
- Treating low perplexity as evidence of factual reliability or safety.