Self-Attention

Mechanism that computes context-aware mixtures of representations; scales well and captures long-range dependencies.

The set of tokens a model can represent; impacts efficiency, multilinguality, and handling of rare strings.

Maximum number of tokens the model can attend to in one forward pass; constrains long-document reasoning.

How many requests or tokens can be processed per unit time; affects scalability and cost.

Architecture based on self-attention and feedforward layers; foundation of modern LLMs and many multimodal models.

Mechanism that computes context-aware mixtures of representations; scales well and captures long-range dependencies.