Neural Networks and Deep Learning Basics
What the exam tests
This domain checks that you can describe a perceptron's computation, explain why activation functions are needed, identify layer types, and outline how training reduces loss through gradient descent and backpropagation.
Core concepts
- Perceptron: computes z = Σ(wᵢxᵢ) + b, then applies an activation. It can separate only linearly separable data - the classic failure on XOR.
- Layers: input layer receives features, hidden layers learn intermediate representations, output layer produces the prediction.
- Activation functions: step (0/1), sigmoid (0 to 1, used for binary output), tanh (−1 to 1), ReLU (max(0, z), standard in hidden layers), softmax (probability distribution over multiple classes).
- Training intuition: forward pass computes output; loss measures error (cross-entropy for classification, MSE for regression); backpropagation computes gradients; gradient descent updates weights in the direction that lowers loss. The learning rate controls step size; an epoch is one full pass over the training data.
Worked example
Perceptron with x₁ = 1, x₂ = 0, w₁ = 0.6, w₂ = 0.4, b = −0.5, step activation. z = (0.6 × 1) + (0.4 × 0) + (−0.5) = 0.1. Since z > 0, output = 1. Reasoning: the bias shifts the decision boundary away from the origin, so even zero inputs do not automatically give zero output. Changing b to −0.8 would flip this result to 0.
Traps that cost marks
- Forgetting the bias term in the z calculation.
- Claiming a single perceptron solves XOR; it cannot, because XOR is not linearly separable.
- Saying activation functions are optional - without non-linearity, stacked layers collapse into one linear function.
- Using softmax for a single binary output; sigmoid is correct there.
- Thinking a very large learning rate trains faster; it usually overshoots and diverges.