Batch norm

Batch-level normalization that rescales activations using statistics pooled across examples, common in convolutional networks but uncommon inside modern transformers.

Batch norm steadies features by using the minibatch's shared mean and variance, which works best when examples line up cleanly but becomes awkward when token streams, batch sizes, or decoding steps vary from one pass to the next.

At a glance

Released

February 2015

Authors

Sergey Ioffe, Christian Szegedy

Optimizes

  • Training Stability

What It Is

Batch normalization computes feature statistics across the minibatch, rescales each feature with those shared numbers, then applies learned scale and shift. One example is partly normalized using information from its batch neighbors rather than only from its own hidden state.

Why It Exists

Batch norm mainly targets training stability in settings where many examples share the same feature layout. It can reduce internal scale drift and make optimization easier when minibatches are large enough to give reliable statistics.

How It Works

For each feature or channel, batch norm measures the minibatch mean and variance, recenters the values, rescales them by the stabilized spread, then applies learned gain and bias. During inference it uses running estimates collected during training rather than the live minibatch.

Math Or Compute Schema

The formula below summarizes one normalized feature channel. The important distinction is the subscript B: the mean and variance come from the batch, not from one token or one hidden vector alone.
Batch normalization
BN(x)=γ⊙x−μBσB2+ϵ+β\mathrm{BN}(x) = \gamma \odot \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} + \beta
xx
Feature values for one channel before normalization.
μB\mu_B
Mean of that feature over the current minibatch.
σB2\sigma_B^2
Variance of that feature over the current minibatch.
ϵ\epsilon
Small constant for numerical stability.
γ\gamma
Learned per-feature scale.
β\beta
Learned per-feature shift.

Compared To Nearby Modules

Batch norm shares statistics across examples. Layer norm and RMSNorm stay inside one token or hidden vector, while group norm stays inside one example but uses smaller channel groups. That difference is why batch norm fits image minibatches better than autoregressive transformer decoding.
Comparison dimensionBatch NormLayer NormGroup Norm
Statistics scopeOne feature or channel across the minibatchAll features inside one token or hidden vectorSmall channel groups inside one example
Depends on batch neighborsYes; each example uses shared batch statisticsNo; each token is normalized on its ownNo; groups stay inside one example
Common useConvolutional and vision models with stable batchesTransformer blocks and sequence modelsVision models when batch norm is awkward at small batch sizes

Example Architectures

Batch norm is common in convolutional backbones and many earlier vision architectures. It appears less often in modern decoder-only language models, so it is useful here mainly as a comparison point.

Limitations And Tradeoffs

Batch norm depends on stable minibatch statistics. Small batches, variable sequence slices, and token-by-token decoding make those statistics noisy or inconvenient, which is why transformer language models usually prefer layer norm or RMSNorm.

Why It Still Matters

Many model families outside LLMs still use batch norm, and papers often compare newer norms against it. Understanding batch norm makes the rest of the normalization family much easier to interpret.

Tags

References

  1. Ioffe, Sergey, and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." arXiv, 2015, https://arxiv.org/abs/1502.03167.

Swish Gated Linear Unit

Previous Page

Group norm

Next Page

On this page

What It IsWhy It ExistsHow It WorksMath Or Compute SchemaCompared To Nearby ModulesExample ArchitecturesLimitations And TradeoffsWhy It Still MattersRelatedTagsReferences