Activation
Activation functions and activation-focused pages across glossary and module docs.
Module type
Module
- Gaussian Error Linear Unit
A smooth activation function that keeps small negative values and is common in transformer feed-forward blocks.
- Hyperbolic Tangent Activation
A smooth activation function that maps each value into a centered -1-to-1 range.
- Leaky Rectified Linear Unit
An activation function that keeps a small negative slope instead of turning every negative value into zero.
- Rectified Linear Unit
A simple activation function that keeps positive values and turns negative values into zero.
- Sigmoid Activation
A smooth activation function that maps each value into a 0-to-1 gate.
- Sigmoid Linear Unit
A smooth activation function that scales each value by its own sigmoid score.