Normalization

Batch-level normalization that rescales activations using statistics pooled across examples, common in convolutional networks but uncommon inside modern transformers.

Channel-group normalization that rescales each example using per-group statistics, often used when batch norm is awkward or unstable.

Per-token mean-and-variance normalization that rescales each hidden vector before the next sublayer in a transformer block.

Query-key normalization that rescales attention queries and keys before their dot products, changing the attention score path rather than the whole residual stream.

Per-token root-mean-square scaling that rescales each hidden vector without mean centering, as used in many modern decoder transformers.

Keeping activations on a predictable scale so deep model stacks stay trainable and their sublayers receive stable inputs.

Layers that rescale hidden vectors inside each transformer block so activations stay stable as depth and batch composition change.