Transformer

Mechanism that computes context-aware mixtures of representations; scales well and captures long-range dependencies.

A parameterized function composed of interconnected units organized in layers with nonlinear activations.

A high-capacity language model trained on massive corpora, exhibiting broad generalization and emergent behaviors.

Converting audio speech into text, often using encoder-decoder or transducer architectures.

Mechanism that computes context-aware mixtures of representations; scales well and captures long-range dependencies.

Attention where queries/keys/values come from the same sequence, enabling token-to-token interactions.

The set of tokens a model can represent; impacts efficiency, multilinguality, and handling of rare strings.

Maximum number of tokens the model can attend to in one forward pass; constrains long-document reasoning.