Early Stopping

Techniques that discourage overly complex solutions to improve generalization (reduce overfitting).

Nonlinear functions enabling networks to approximate complex mappings; ReLU variants dominate modern DL.

Activation max(0, x); improves gradient flow and training speed in deep nets.

Gradients shrink through layers, slowing learning in early layers; mitigated by ReLU, residuals, normalization.

Gradients grow too large, causing divergence; mitigated by clipping, normalization, careful init.

A gradient method using random minibatches for efficient training on large datasets.

Controls the size of parameter updates; too high diverges, too low trains slowly or gets stuck.

Number of samples per gradient update; impacts compute efficiency, generalization, and stability.

One complete traversal of the training dataset during training.