Token-to-probability chain

The numeric output of a layer after its linear transform and nonlinearity, passed forward through the network.

The backward pass that applies the chain rule on a computational graph to compute gradients for every differentiable parameter.

A record of which operations produced each tensor so automatic differentiation can run backward.

A dense vector that represents a token or other discrete item so the model can run continuous math on it.

A measure of how spread out or uncertain a probability distribution is over next-token choices.

The direction and size of change each parameter should take to reduce the training loss.

The width of a model's internal vectors—the number of dimensions in each token embedding and each token's per-position hidden state before the vocabulary projection.

A raw, unnormalized score for each vocabulary item before softmax turns scores into probabilities.

A scalar score that measures how wrong the model's predictions are so training knows which direction to improve.

Extra memory an adaptive optimizer keeps beyond model weights, such as momentum and variance estimates for each parameter.

A learnable numeric value stored in the model weights that training updates to fit data.

A function that turns a vector of logits into a probability distribution that sums to one.

A positive scaling factor applied to logits before softmax to make the next-token distribution sharper or flatter.

A multi-dimensional array of numbers that carries activations, weights, or gradients through the network.

The smallest unit of text a language model reads and predicts—usually a word piece, not always a whole word. Each token ID maps to a dense vector through vector embedding of model hidden size before attention runs.

An ordered list of numbers that represents a point or direction in continuous space—embeddings and activations are vectors at different stages of the model.