Rectified Linear Unit
A simple activation function that keeps positive values and turns negative values into zero.
The rectified linear unit (ReLU) is a simple activation function that leaves positive values alone and turns negative values into zero. It became important because deep networks needed a cheap way to add nonlinearity without the training slowdown that older saturating activations such as sigmoid or tanh often caused.
At a glance
Released
January 2010
Authors
Vinod Nair, Geoffrey E. Hinton
Optimizes
- Activation Sparsity
What It Is
The rectified linear unit (ReLU) is an activation function used in many neural networks. It applies one simple rule to each value: keep positive numbers and replace negative numbers with zero. In transformer feed-forward layers, that rule decides which hidden features stay active before the layer projects back to the model width.Why It Exists
ReLU exists because neural networks need nonlinearity to model the curved, conditional patterns that show up in real data. Without an activation rule like this, stacked linear layers still collapse into one linear transform. ReLU also became popular because it gave that nonlinearity in a very cheap form that often trained more easily than older saturating activations such as sigmoid or tanh, helping deeper models reduce training loss more reliably.How It Works
ReLU works one value at a time. If the value is positive, it stays as it is. If the value is negative, it becomes zero. In a transformer feed-forward layer, the model first expands into a wider hidden state, applies that rule to each hidden value, and then projects the filtered result back down.Activation Curves
f(x)
x
- ReLU
Activation comparison chart showing how rectified linear unit, leaky rectified linear unit, and sigmoid linear unit treat positive and negative values
Hidden State Activity Through ReLU
Hidden State Before ReLU
-303
Hidden State After ReLU
-303
Two heatmaps compare a hidden state before rectified linear unit and after rectified linear unit, showing negative activations removed while positive activations remain.
Math Or Compute Schema
The formula below shows the ReLU rule for one value at a time. The key detail is the hard cutoff at zero.- One input value before the activation rule is applied.
Compared To Nearby Modules
ReLU is the sharpest cutoff in this activation family. Leaky rectified linear unit keeps a faint negative path, while sigmoid linear unit smooths the whole transition instead of using a hard corner.| Comparison dimension | ReLU | LeakyReLU | SiLU |
|---|---|---|---|
| Negative branch | Clamp every negative hidden value to 0 | Keep a small slope alpha x for x < 0 | Shrink negative values smoothly with x sigma(x) |
| Transition shape | Hard corner at 0 | Hard corner with a weak negative path | Smooth curve through 0 |
| Main tradeoff | Very simple and sparse, but it throws away all negative signal | Preserves gradient on the negative side, but stays less smooth than SiLU | Smoother hidden filtering, but less minimal than ReLU |