Deep Learning and Neural Networks
Anatomy
A neuron computes a weighted sum of inputs plus a bias, then applies a nonlinear activation. Stacked layers form a network: input layer, hidden layers, output layer. Weights are learned; architecture, learning rate and batch size are hyperparameters you set.
Activations and loss
ReLU is the default for hidden layers (cheap, avoids saturation). Sigmoid outputs a probability for binary tasks and for independent multi-label classes. Softmax outputs a distribution across mutually exclusive classes - the outputs sum to 1. Regression heads use a linear activation. Losses pair with tasks: cross-entropy for classification, MSE/MAE for regression. The loss is what training minimizes; accuracy is only a reporting metric.
Training loop
Forward pass produces predictions, the loss compares them to labels, backpropagation computes gradients via the chain rule, and an optimizer (SGD, Adam) updates weights scaled by the learning rate. One pass over the data is an epoch; a batch is the subset processed per update. A learning rate too high diverges or yields NaN loss; too low trains slowly or stalls.
Architectures to recognize
CNNs use shared convolution filters plus pooling for grid data such as images. RNNs/LSTMs handle sequences and suffer vanishing gradients on long inputs. Transformers replace recurrence with self-attention plus positional encodings, enabling parallel training and long-range context - the architecture behind modern LLMs. Embeddings map discrete tokens or categories to dense vectors where distance is meaningful.
Traps
'More layers always help' is false. Softmax is wrong for multi-label problems - use independent sigmoids. Dropout is disabled at inference. Deep learning is not automatically better than classical ML on small tabular datasets.
Rule of thumb
Pick the output activation from the target: linear for regression, sigmoid for binary or multi-label, softmax for single-label multiclass.