Layer norm
Per-token mean-and-variance normalization that rescales each hidden vector before the next sublayer in a transformer block.
Layer norm keeps each token's hidden vector in a predictable range by using that token's own feature mean and variance, which makes it a clean fit for transformer blocks that cannot rely on stable minibatch statistics.
At a glance
Optimizes
- Training Stability
What It Is
Layer normalization rescales one token's hidden vector by subtracting its feature mean, dividing by a stabilized feature standard deviation, then applying learned scale and shift. The statistics come from that one vector, not from the rest of the batch.Why It Exists
Layer norm targets training stability in deep sequence models where batch statistics are either noisy or structurally inconvenient. It keeps residual-stream magnitudes from drifting too far before attention or feed-forward sublayers run.How It Works
For one token position, layer norm measures the mean and variance across the hidden features in that vector, recenters and rescales the vector, then applies learned gain and bias. The switcher below keeps that path next to RMSNorm so you can see that the extra mean-centering step is the main structural difference.Token state xFeature mean μ and variance σ²Subtract mean and divide by standard deviationApply learned gain γ and bias βStable token for next sublayerMean centering makes the output zero-centered before the learned affine stepStatistics come from this one token vector, not from the batchToken state x to Feature mean μ and variance σ²Feature mean μ and variance σ² to Subtract mean and divide by standard deviationSubtract mean and divide by standard deviation to Apply learned gain γ and bias βApply learned gain γ and bias β to Stable token for next sublayer
Math Or Compute Schema
These two formulas apply to one token vector at a time. Layer norm subtracts the feature mean before scaling, while RMSNorm keeps only the scale correction.Compared To Nearby Modules
Layer norm stays within one token, unlike batch norm, and it includes mean centering, unlike RMSNorm. Group norm also avoids batch coupling, but it normalizes channel groups rather than an entire token vector.| Comparison dimension | Layer Norm | RMSNorm | Batch Norm |
|---|---|---|---|
| Mean centering | Yes; subtracts the token's feature mean | No; scales by root mean square only | Yes; uses the minibatch feature mean |
| Statistics scope | One token or hidden vector at a time | One token or hidden vector at a time | One feature or channel across the minibatch |
| Fit for transformer blocks | Very common and structurally simple | Very common in newer decoder-only LLMs | Usually awkward for autoregressive token streams |