ReLU

Nonlinear functions enabling networks to approximate complex mappings; ReLU variants dominate modern DL.

Gradients shrink through layers, slowing learning in early layers; mitigated by ReLU, residuals, normalization.

Gradients grow too large, causing divergence; mitigated by clipping, normalization, careful init.

Methods to set starting weights to preserve signal/gradient scales across layers.

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.

Number of samples per gradient update; impacts compute efficiency, generalization, and stability.

One complete traversal of the training dataset during training.

Halting training when validation performance stops improving to reduce overfitting.

Nonlinear functions enabling networks to approximate complex mappings; ReLU variants dominate modern DL.