Sigmoid Activation
A smooth activation function that maps each value into a 0-to-1 gate.
The sigmoid activation, also called the logistic sigmoid, turns any input value into a number between 0 and 1. It was useful in earlier neural networks because that output can act like a soft gate or probability-like score, but its flat tails make deep networks train slowly when many values saturate.
At a glance
Released
October 1986
Authors
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams
Optimizes
- Smooth Gating
What It Is
The sigmoid activation is a smooth function that maps each input value x to a value between 0 and 1. Large negative inputs move close to 0, large positive inputs move close to 1, and inputs near 0 land near 0.5. In a neural network, that shape can turn a raw hidden value into a soft gate.Why It Exists
Sigmoid became common because early neural networks needed a differentiable nonlinear rule. A stack of linear layers without activation functions still behaves like one linear layer. Sigmoid added a smooth bend and gave outputs that looked easy to interpret as on-or-off scores.How It Works
Sigmoid works one value at a time. It exponentiates the negative input, adds 1, and takes the reciprocal. Near x = 0, small input changes still move the output. Far from 0, the curve flattens near 0 or 1, so very large positive or negative values change the output only a little.Activation Curves
f(x)
x
- Sigmoid
Activation chart showing the sigmoid curve bounded between 0 and 1 with flat tails for large negative and positive inputs
Math Or Compute Schema
The formula below shows the logistic sigmoid rule. The important feature is the bounded output range and the flat tails on both sides.- One input value before the activation rule is applied.
- Base of the natural exponential.
Compared To Nearby Modules
Compared with ReLU and SiLU, sigmoid is the most strongly bounded curve. ReLU keeps growing for positive inputs, SiLU uses sigmoid as a smooth self-gate, and sigmoid by itself compresses every value into the same 0-to-1 range.| Comparison dimension | Sigmoid | ReLU | SiLU |
|---|---|---|---|
| Output range | Always between 0 and 1 | 0 for negative values, unbounded for positive values | Unbounded on the positive side with smooth negative shrinkage |
| Curve shape | Smooth S-curve with flat tails | Hard corner at 0 | Smooth self-gated curve through 0 |
| Main tradeoff | Useful as a gate, but saturated tails weaken gradients | Cheap and strong on the positive side, but drops negative signal | Keeps sigmoid-like smooth gating while preserving a stronger positive path |
Example Architectures
Sigmoid appears in older dense networks and still appears inside gates, probabilities, and binary-output heads. Modern transformer feed-forward blocks usually avoid using plain sigmoid as the main hidden activation because saturation can weaken gradient flow through deep stacks.Limitations And Tradeoffs
Sigmoid saturation is the main tradeoff. Once an input lands far into either tail, the output is almost fixed near 0 or 1, and the local gradient becomes small. That can make early layers learn slowly in deep networks, especially compared with activations whose positive side keeps a stronger slope.Why It Still Matters
Sigmoid still matters because it teaches the idea of a smooth gate. Understanding why it saturates also explains why later activation functions, including ReLU and SiLU, changed the curve shape to keep useful signal moving through deep networks.References
- Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning Representations by Back-Propagating Errors." Nature, vol. 323, 1986, pp. 533-536, https://www.semanticscholar.org/paper/Learning-representations-by-back-propagating-errors-Rumelhart-Hinton/052b1d8ce63b07fec3de9dbb583772d860b7c769.
- Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016, https://www.deeplearningbook.org/.