Depth vs Width

Prevents attention to future tokens during training/inference.

A single attention mechanism within multi-head attention.

Allows model to attend to information from different subspaces simultaneously.

Stores past attention states to speed up autoregressive decoding.

A narrow hidden layer forcing compact representations.

Using same parameters across different parts of a model.

Built-in assumptions guiding learning efficiency and generalization.

The range of functions a model can represent.