RMSNorm
Per-token root-mean-square scaling that rescales each hidden vector without mean centering, as used in many modern decoder transformers.
RMSNorm keeps each token's hidden vector in a stable scale band by dividing by its root mean square instead of using full mean-and-variance normalization, which makes it a lighter fit for many modern decoder-only transformer stacks.
At a glance
Optimizes
- Training Stability
- Inference Efficiency
What It Is
Root mean square normalization rescales one token's hidden vector by dividing by the root mean square of its features, then applying learned gain. Unlike layer norm, it does not subtract the feature mean first.Why It Exists
RMSNorm targets stable hidden-state scale with less normalization work than full layer norm. It keeps per-token magnitudes predictable while avoiding the mean-centering step.How It Works
For one token vector, RMSNorm computes the root mean square over the features, divides the vector by that stabilized scale, then applies learned gain. The switcher below keeps that path next to layer norm so you can see that RMSNorm preserves the same per-token pattern while skipping mean subtraction.Token state xRoot mean square over featuresDivide by RMS scale onlyApply learned gain γStable token for next sublayerNo mean subtraction step: RMSNorm corrects magnitude without recentering the vectorStatistics still come from one token vector, not from the batchToken state x to Root mean square over featuresRoot mean square over features to Divide by RMS scale onlyDivide by RMS scale only to Apply learned gain γApply learned gain γ to Stable token for next sublayer
Math Or Compute Schema
These two formulas both normalize one token vector at a time. RMSNorm keeps the scale correction from layer norm but removes the mean-centering term.Compared To Nearby Modules
RMSNorm is closest to layer norm because both normalize one token at a time. The main difference is that RMSNorm keeps the scale correction but drops mean centering and often omits the learned bias term as well.| Comparison dimension | RMSNorm | Layer Norm | Batch Norm |
|---|---|---|---|
| Mean centering | No; scales by root mean square only | Yes; subtracts the token's feature mean | Yes; uses minibatch feature statistics |
| Learned shift term | Often omitted; learned gain is the main parameter | Usually includes both learned gain and bias | Usually includes both learned gain and bias |
| Common placement | Per-token norm inside modern decoder transformers | Per-token norm around transformer sublayers | Feature or channel norm across image-style minibatches |