Neural Networks and Deep Learning
Core concepts
A neural network is layers of neurons (weighted sums plus a nonlinear activation) trained by backpropagation and gradient descent. Know that nonlinearity is what lets stacked layers learn complex functions; without it, depth collapses to a linear model. Understand weights and biases, loss functions, learning rate (too high diverges, too low stalls), epochs, batches, and regularization (dropout, weight decay, early stopping).
Architectures
Map architecture to data type: CNNs for grids like images (local filters, weight sharing); RNNs/LSTMs for sequences (with vanishing-gradient limits); transformers for sequences via self-attention, now dominant in language and increasingly in vision. Know embeddings as learned dense vectors capturing similarity.
Traps
Questions conflate parameters (learned weights) with hyperparameters (learning rate, layer count-set before training). Another trap: assuming deeper is always better; depth raises capacity and overfitting risk and needs more data and compute. Watch for confusion between training loss and validation loss-diverging curves signal overfitting. Also remember GPU/TPU acceleration is an efficiency fact, not a learning algorithm.
Rule of thumb
Diagnose with curves: if training loss falls while validation loss rises, you are overfitting; if both stay high, you are underfitting. Fix the first with regularization and more data, the second with more capacity or better features.
Worked example
An image classifier uses a CNN because convolution exploits spatial locality and shares weights across positions, drastically cutting parameters versus a fully connected network on the same pixels.