Deep Learning and Neural Networks
What to know
Expect backpropagation and the chain rule, activation functions (ReLU and its variants, sigmoid, tanh, softmax), loss functions (cross-entropy, MSE), optimizers (SGD, momentum, Adam), regularization (dropout, weight decay, early stopping, batch/layer normalization), initialization, vanishing/exploding gradients, and architecture families: CNNs (convolution, pooling, receptive field), RNNs/LSTMs, and Transformers (self-attention, positional encoding). Understand overfitting signals and hyperparameter effects.
Traps
- Thinking dropout is active at inference - it is disabled then, with scaling compensation.
- Assuming batch norm behaves identically in train and eval modes.
- Believing more layers always help; degradation and vanishing gradients occur.
- Mixing up convolution parameter sharing (translation equivariance) with fully connected layers.
- Forgetting softmax + cross-entropy is the standard classification pairing.
Rule of thumb
Diagnose with learning curves: train loss low and val loss rising means overfit (add regularization, data, or reduce capacity); both high means underfit (increase capacity or train longer).
Worked example
A conv layer with 3×3 kernels, 64 input channels, 128 output channels has 3·3·64·128 + 128 = 73,856 parameters - weights plus one bias per output channel. Parameter-count questions reward careful arithmetic.