Next-Token Prediction

Generates sequences one token at a time, conditioning on past tokens.

Predicts masked tokens in a sequence, enabling bidirectional context; often used for embeddings rather than generation.

Achieving task performance by providing a small number of examples inside the prompt without weight updates.

Stepwise reasoning patterns that can improve multi-step tasks; often handled implicitly or summarized for safety/priv...

Randomly zeroing activations during training to reduce co-adaptation and overfitting.

An RNN variant using gates to mitigate vanishing gradients and capture longer context.

Injects sequence order into Transformers, since attention alone is permutation-invariant.

Converting text into discrete units (tokens) for modeling; subword tokenizers balance vocabulary size and coverage.