Neural Networks and Deep Learning
What this domain tests
Whether you understand what happens inside a network during forward and backward passes, and whether you can diagnose a training run that is diverging, stalling, or overfitting from the symptoms it shows.
Core rules
- A layer computes z = Wx + b, then a = f(z); backpropagation applies the chain rule to get gradients for every parameter.
- Activations: ReLU family in hidden layers, sigmoid for binary and softmax for multiclass outputs, tanh still common in some recurrent designs.
- Vanishing and exploding gradients are mitigated by ReLU-style activations, He or Xavier initialisation, batch normalisation, residual connections, and gradient clipping.
- Regularise with dropout, weight decay, early stopping, and data augmentation.
- Optimisers: SGD with momentum often generalises well; Adam and AdamW converge faster and are the usual default for transformers.
- Learning rate is the highest-leverage hyperparameter. Too high diverges or produces NaNs; too low stalls progress.
- Architecture follows data shape: CNNs for spatial grids, sequence models for ordered data, transformers for long-range dependencies and multimodality.
Worked example
A binary classifier's loss sits at 0.69 and never moves. Since ln(2) is about 0.69, the network is outputting 0.5 everywhere - it has learned nothing. Likely causes, in order: dead ReLU units from an aggressive learning rate, a learning rate so small that updates vanish, or misaligned features and labels. Check label alignment and gradient magnitudes first; only then restructure the architecture.
Traps that cost marks
Reporting accuracy on imbalanced softmax output. Forgetting to switch to evaluation mode, so dropout stays active at inference. Reusing training-set normalisation statistics computed from the test set. Blaming the architecture when the learning rate is simply wrong.