Core Machine Learning and AI Knowledge
What to know
A decoder-only transformer maps token IDs to embeddings, injects positional information (RoPE, ALiBi, or learned tables), and passes them through N blocks of causally masked self-attention plus a feed-forward network. In attention, queries, keys, and values are linear projections; scores QK^T are scaled by 1/sqrt(d_head) before masking and softmax. Multi-head attention splits d_model into H heads of size d_model/H, so H must divide d_model. Modern stacks use pre-norm RMSNorm, SwiGLU FFNs, and grouped-query attention (GQA) to shrink the KV cache relative to full multi-head attention.
Tokenization is a separate artifact: BPE or SentencePiece vocabularies determine sequence length and cost, so parameter count alone does not predict compute - FLOPs scale with params and tokens.
Know the adaptation families: full fine-tuning, LoRA (frozen base plus low-rank A/B matrices, update scaled by alpha/r), QLoRA (quantized frozen base), SFT, RLHF (reward model + PPO), and DPO (direct preference optimization, no reward model). Quantization (FP8, INT8, INT4 via GPTQ/AWQ) shrinks weight memory; the KV cache instead scales with batch x sequence length.
Traps
- Confusing temperature (reshapes the distribution) with top-p/top-k (truncates it).
- Forgetting the factor of 2 for K and V in KV-cache arithmetic.
- Assuming rank r alone sets LoRA strength - alpha/r controls the magnitude.
- Comparing perplexity across models with different tokenizers.
Worked example
KV cache for 32 layers, 32 heads, head_dim 128, FP16 (2 bytes), batch 1, 4096 tokens: 2 (K,V) x 32 x 32 x 128 x 4096 x 2 bytes = ~2 GiB. Double the sequence length and you double cache memory, often before weights become the bottleneck.