Gaussian Error Linear Unit
A smooth activation function that keeps small negative values and is common in transformer feed-forward blocks.
The Gaussian Error Linear Unit, usually shortened to GELU, scales each input by how likely it is to be useful under a Gaussian-shaped gate. That gives transformer feed-forward blocks a smoother transition than rectified linear unit (ReLU), keeps some small negative values, and avoids the hard cutoff that classic ReLU uses.
At a glance
Optimizes
- Smooth Gating
What It Is
The Gaussian Error Linear Unit, usually shortened to GELU, is an activation function that keeps an input value x, shrinks it, or nearly removes it depending on where x falls on a smooth Gaussian-shaped curve. Large positive values pass through strongly, values near 0 are only partly kept, and small negative values can still survive instead of being cut off at exactly 0.Why It Exists
GELU exists because a hard on-or-off gate is not always the best filter for deep hidden states. ReLU keeps every positive value and drops every negative value. GELU instead treats the gate as smooth and probabilistic, so values near 0 are not forced into a sharp decision boundary.How It Works
GELU works one value at a time. It multiplies x by a Gaussian cumulative score Phi(x). Large positive values get a score near 1, so they mostly pass through. Large negative values get a score near 0, so they mostly disappear. Values near 0 are only partly passed, which makes the curve look like a softened version of ReLU. In transformer feed-forward blocks, that smoother gate is often used after the first wide projection and before projecting back to the model width.Activation Curves
f(x)
x
- GELU
- ReLU
- SiLU
Activation chart comparing GELU, ReLU, and SiLU, showing GELU as a smooth ReLU-like curve that keeps small negative values
Math Or Compute Schema
The formula below shows the exact GELU rule. The important idea is that x is scaled by the Gaussian cumulative distribution Phi(x), which creates a smooth gate instead of a hard sign check.- One input value before the activation rule is applied.
- Gaussian Error Linear Unit activation.
- Cumulative distribution function of the standard normal distribution.
Compared To Nearby Modules
Compared with ReLU, GELU keeps the same broad idea of favoring positive values but replaces the hard cutoff with a smooth curve. Compared with SiLU, GELU is another smooth self-scaling activation, but the gate comes from the Gaussian cumulative distribution rather than sigmoid. That smoother shape is one reason transformer feed-forward blocks often prefer GELU over plain ReLU.| Comparison dimension | GELU | ReLU | SiLU |
|---|---|---|---|
| Negative branch | Keeps some small negative values with smooth shrinkage | Drops every negative value to 0 | Shrinks negative values smoothly with x sigma(x) |
| Transition shape | Smooth ReLU-like curve with no hard corner | Piecewise-linear with a hard corner at 0 | Smooth self-gated curve through 0 |
| Common use | Common dense activation in transformer feed-forward blocks | Classic cheap baseline for dense hidden layers | Smooth activation and gate inside SwiGLU-style feed-forward variants |