Sigmoid Linear Unit
A smooth activation function that scales each value by its own sigmoid score.
The sigmoid linear unit (SiLU) smoothly scales each value by its own sigmoid score, making it a softer alternative to ReLU and an important stepping stone to gated designs such as SwiGLU.
At a glance
Released
February 2017
Authors
Stefan Elfwing, Eiji Uchibe, Kenji Doya
Optimizes
- Activation Smoothness
What It Is
The sigmoid linear unit (SiLU) is an activation function that multiplies each value x by sigmoid(x). Large positive values pass through strongly, values near zero are softened, and negative values are reduced smoothly instead of being cut off sharply. In transformer feed-forward layers, that gives the model a softer filter than ReLU.Why It Exists
SiLU optimizes for a smoother activation rule. Instead of a hard cutoff, it lets the negative side fade gradually, which often fits modern wide feed-forward layers better than a sharp zero clamp.How It Works
SiLU works one value at a time by multiplying that value by its own sigmoid score. Large positive values mostly survive, small values are softened, and negative values shrink rather than snapping to zero. In a transformer feed-forward layer, the model applies that rule after expanding into a wider hidden state and before projecting back down.Activation Curves
f(x)
x
- ReLU
- SiLU
Activation comparison chart showing how rectified linear unit, leaky rectified linear unit, and sigmoid linear unit reshape positive and negative values
Math Or Compute Schema
The formula below shows the smooth self-gating rule directly. The sigmoid term is what turns one value into its own soft gate.- One input value before the activation rule is applied.
- Sigmoid function applied to that same input value.
Compared To Nearby Modules
Compared with ReLU and LeakyReLU, SiLU is the smooth option in this activation family. It still stays inside one dense feed-forward path, but it prepares the reader for the stronger two-branch gate used by SwiGLU.| Comparison dimension | SiLU | ReLU | LeakyReLU |
|---|---|---|---|
| Negative branch | Shrink negative values smoothly with x sigma(x) | Clamp every negative hidden value to 0 | Keep a small slope alpha x for x < 0 |
| Transition shape | Smooth curve through 0 | Hard corner at 0 | Hard corner with a weak negative path |
| Main tradeoff | Smoother feature filtering, but more complex than the ReLU family baseline | Simple and sparse, but fully drops negative signal | Keeps some negative gradient, but still uses a piecewise-linear cutoff |