Hyperbolic Tangent Activation
A smooth activation function that maps each value into a centered -1-to-1 range.
The hyperbolic tangent activation, usually called tanh, maps any input value into a number between -1 and 1. It kept the smooth S-curve behavior of sigmoid while centering outputs around 0, but its flat tails still made deep networks train slowly when many values saturated.
At a glance
Released
October 1986
Authors
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams
Optimizes
- Centered Activation
What It Is
The hyperbolic tangent activation is a smooth function that maps each input value x to a value between -1 and 1. Large negative inputs move close to -1, large positive inputs move close to 1, and inputs near 0 stay near 0. In a neural network, that centered range lets a hidden value carry negative, neutral, or positive evidence.Why It Exists
Tanh was useful because early neural networks needed differentiable nonlinear rules, but sigmoid outputs are always positive. Centering the output around 0 often makes the next layer easier to balance because positive and negative activations can offset each other.How It Works
Tanh works one value at a time. Near x = 0, the curve has a strong slope and behaves almost like the identity function. Far from 0, the curve flattens near -1 or 1, so very large negative or positive values change the output only a little.Activation Curves
f(x)
x
- Tanh
Activation chart showing the tanh curve centered at 0 and bounded between -1 and 1 with flat tails on both sides
Math Or Compute Schema
The formula below shows the centered S-curve rule. The key features are the -1-to-1 output range, the zero-centered midpoint, and the flat tails on both sides.- One input value before the activation rule is applied.
- Hyperbolic tangent function.
- Base of the natural exponential.
Compared To Nearby Modules
Compared with sigmoid, tanh keeps the same saturating S-curve shape but shifts the midpoint from 0.5 to 0. Compared with ReLU, tanh is bounded on both sides, so it can express negative activations but cannot keep growing for large positive inputs.| Comparison dimension | Tanh | Sigmoid | ReLU |
|---|---|---|---|
| Output range | Always between -1 and 1 | Always between 0 and 1 | 0 for negative values, unbounded for positive values |
| Center point | Centered at 0 | Centered at 0.5 | Hard cutoff at 0 |
| Main tradeoff | Keeps negative and positive evidence, but saturated tails weaken gradients | Useful as a soft gate, but positive-only outputs can shift the next layer | Keeps a strong positive slope, but drops negative signal |
Example Architectures
Tanh appears in older dense networks and recurrent neural networks, especially where a hidden state benefits from a bounded centered range. Modern transformer feed-forward blocks usually use ReLU-like or gated activations instead because tanh saturation can weaken gradient flow through deep stacks.Limitations And Tradeoffs
The main tradeoff is saturation. Once an input lands far into either tail, the output is almost fixed near -1 or 1, and the local gradient becomes small. That can slow learning in early layers of deep networks, even though the centered output is easier to work with than sigmoid's positive-only range.Why It Still Matters
Tanh still matters because it separates two ideas that are easy to confuse: centered activations can help signal balance, while saturating tails can still make optimization harder. That contrast explains why later activation functions kept stronger slopes while moving away from bounded S-curves.References
- Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning Representations by Back-Propagating Errors." Nature, vol. 323, 1986, pp. 533-536, https://www.semanticscholar.org/paper/Learning-representations-by-back-propagating-errors-Rumelhart-Hinton/052b1d8ce63b07fec3de9dbb583772d860b7c769.
- Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016, https://www.deeplearningbook.org/.