Causal Mask

A single attention mechanism within multi-head attention.

Allows model to attend to information from different subspaces simultaneously.

Stores past attention states to speed up autoregressive decoding.

Encodes positional information via rotation in embedding space.

Using same parameters across different parts of a model.

Built-in assumptions guiding learning efficiency and generalization.

The range of functions a model can represent.

Tradeoffs between many layers vs many neurons per layer.