Second-Order Methods

Matrix of second derivatives describing local curvature of loss.

Neural networks can approximate any continuous function under certain conditions.

Allows gradients to bypass layers, enabling very deep networks.

Early architecture using learned gates for skip connections.

A wide basin often correlated with better generalization.

Adjusting learning rate over training to improve convergence.

Gradually increasing learning rate at training start to avoid divergence.

Limiting gradient magnitude to prevent exploding gradients.