Vanishing Gradient

Gradients grow too large, causing divergence; mitigated by clipping, normalization, careful init.

Methods to set starting weights to preserve signal/gradient scales across layers.

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.

Randomly zeroing activations during training to reduce co-adaptation and overfitting.

One complete traversal of the training dataset during training.

Halting training when validation performance stops improving to reduce overfitting.

Nonlinear functions enabling networks to approximate complex mappings; ReLU variants dominate modern DL.

Activation max(0, x); improves gradient flow and training speed in deep nets.