Universal Approximation Theorem

Allows gradients to bypass layers, enabling very deep networks.

Early architecture using learned gates for skip connections.

A narrow hidden layer forcing compact representations.

Using same parameters across different parts of a model.

Gradually increasing learning rate at training start to avoid divergence.

Limiting gradient magnitude to prevent exploding gradients.

Optimization using curvature information; often expensive at scale.

Matrix of second derivatives describing local curvature of loss.