Leaky Rectified Linear Unit
An activation function that keeps a small negative slope instead of turning every negative value into zero.
The leaky rectified linear unit (LeakyReLU) keeps a small negative slope instead of zeroing every negative value, so weak negative evidence and gradients can still pass through.
At a glance
Released
January 2013
Authors
Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng
Optimizes
- Gradient Flow
What It Is
The leaky rectified linear unit (LeakyReLU) is a small variation on the rectified linear unit. Positive values pass through as usual, but negative values are multiplied by a small constant such as 0.01 instead of becoming zero. In feed-forward layers, that means the model can keep a weak negative signal instead of shutting it off completely.Why It Exists
LeakyReLU optimizes for better negative-side gradient flow while staying very close to ReLU. It keeps the same cheap piecewise-linear shape, but it avoids a fully dead negative branch.How It Works
LeakyReLU works one value at a time. Positive values keep their size. Negative values stay negative, but they are scaled down by a small factor. In a transformer feed-forward layer, the model expands into a wider hidden state, applies that rule to each hidden value, and then projects the result back down.Token state h_tWide projection W_1LeakyReLU(x) = max(alpha x, x)Negative values still flow, but only through a small slope alpha xProject back with W_2Updated token stateToken state h_t to Wide projection W_1Wide projection W_1 to LeakyReLU(x) = max(alpha x, x)LeakyReLU(x) = max(alpha x, x) to Project back with W_2Project back with W_2 to Updated token state
Math Or Compute Schema
The formula below shows the same split as ReLU, but the negative branch no longer vanishes. Instead, it keeps a small slope alpha.- One input value before the activation rule is applied.
- Small negative-side slope, often around 0.01.
Compared To Nearby Modules
LeakyReLU sits between ReLU and SiLU. It is less abrupt than ReLU because the negative side survives, but it still keeps a piecewise-linear corner instead of moving to the fully smooth SiLU curve.| Comparison dimension | LeakyReLU | ReLU | SiLU |
|---|---|---|---|
| Negative branch | Keep a small slope alpha x for x < 0 | Clamp every negative hidden value to 0 | Shrink negative values smoothly with x sigma(x) |
| Transition shape | Hard corner with a weak negative path | Hard corner at 0 | Smooth curve through 0 |
| Main tradeoff | Improves negative-side gradient flow, but remains less smooth than SiLU | Simplest rule, but fully drops negative signal | Smoother hidden filtering, but less minimal than a piecewise-linear rule |