Deep Learning and Neural Network Architectures
Domain scope
This domain tests architecture-level reasoning: tensor shapes, parameter counts, activation and normalization choices, and why training diverges. Questions often ask you to trace a tensor through a CNN block or explain attention cost.
Core rules and concepts
- A dense layer computes y = σ(Wx + b); backpropagation applies the chain rule and stores activations for gradient computation.
- Activations: ReLU is cheap but can die; GELU/SiLU are common in Transformers; sigmoid is for independent binary outputs, softmax for mutually exclusive classes.
- Normalization: BatchNorm uses batch statistics in training and running statistics in inference; LayerNorm normalizes per sample and is standard in Transformers.
- CNN: convolution shares weights across spatial positions; output size = floor((H + 2P − K)/S) + 1.
- Transformer self-attention costs O(n²·d) in sequence length n, which is why long-context inference is memory-hungry.
Worked example
Input 3×224×224 passes a Conv2d with 64 filters, kernel 3×3, stride 2, padding 1. Output height = floor((224 + 2 − 3)/2) + 1 = 112, so output is 64×112×112. Parameters = (3×3×3)×64 + 64 biases = 1,792. A following 2×2 max-pool with stride 2 yields 64×56×56. Forgetting the bias gives 1,728 and is a common wrong answer.
Traps that cost marks
- Mixing NCHW and NHWC when computing shapes or converting models.
- Using BatchNorm's training behavior during inference, or forgetting to switch model.eval().
- Applying softmax before a loss that already includes softmax (e.g., cross-entropy with logits).
- Assuming attention memory scales linearly with sequence length.