Group norm
Channel-group normalization that rescales each example using per-group statistics, often used when batch norm is awkward or unstable.
Group norm keeps normalization inside one example by measuring small channel groups instead of the whole minibatch, which helps when batch sizes are too small or too irregular for batch norm to behave well.
At a glance
Optimizes
- Small Batch Stability
What It Is
Group normalization splits channels into groups, computes a mean and variance within each group for one example, normalizes those channels, then applies learned scale and shift. It avoids using minibatch-wide statistics while still sharing information across related channels.Why It Exists
Group norm targets stable normalization when minibatch statistics are unreliable. It is especially helpful in models where memory limits force small batches or where example shapes vary too much for batch-wide estimates to stay steady.How It Works
The module partitions channels into fixed groups, measures mean and variance inside each group for one example, recenters and rescales the group, then applies learned gain and bias. Each example is normalized independently, but the groups preserve some local channel structure.Input feature mapSplit channels into groupsGroup mean and varianceNormalize each groupLearned scale and shiftOutput feature mapInput feature map to Split channels into groupsSplit channels into groups to Group mean and varianceGroup mean and variance to Normalize each groupNormalize each group to Learned scale and shiftLearned scale and shift to Output feature map
Math Or Compute Schema
The formula below describes one group inside one example. The group index matters because the statistics are shared only within that channel subset.- Feature values in group g for one example.
- Channel-group index.
- Mean of the features in group g.
- Variance of the features in group g.
- Small constant for numerical stability.
- Learned per-feature scale.
- Learned per-feature shift.
Compared To Nearby Modules
Group norm sits between batch norm and layer norm. Like layer norm, it stays inside one example. Like batch norm, it can still share statistics across multiple channels, just within a smaller local group.| Comparison dimension | Group Norm | Batch Norm | Layer Norm |
|---|---|---|---|
| Statistics scope | Channel groups inside one example | One feature or channel across the minibatch | All features inside one token or hidden vector |
| Depends on batch neighbors | No; groups stay within one example | Yes; examples share live batch statistics | No; one token is normalized on its own |
| Common use | Vision models with small or irregular batches | Convolutional models with stable batch structure | Transformer blocks and sequence models |