Activation
The nonlinear step that reshapes a layer's output so stacked layers can model richer patterns than repeated linear transforms alone.
An activation is the value a hidden layer passes forward after a learned linear transform and, commonly, an activation function. That nonlinear step lets the model keep, dampen, or reshape signals before the next layer. Without those steps, stacked linear layers collapse into another linear transform and cannot build the same kind of intermediate features people expect from deep networks.
What It Is
An activation is the value a layer passes forward after a learned linear transform and, commonly, an activation function such as rectified linear unit (ReLU) or sigmoid linear unit (SiLU). The linear transform scales and mixes the incoming features; the activation function then applies a simple nonlinear rule point by point across those hidden values. In plain language, the weights produce a draft signal and the activation decides how strongly each part should survive before the next layer reads it.Why Nonlinear Steps Matter
Without nonlinearity, stacked linear layers collapse into another linear transform. Each layer can still scale and combine features, but the stack as a whole cannot bend the signal in the ways deep models rely on. Nonlinear activation functions break that collapse: they let later layers respond differently to different ranges of input, create thresholds and smooth gates, and build intermediate features that a single linear map could not express on its own.Where Readers Meet It
Readers most often meet activations inside feed-forward networks, where each token state is expanded, filtered through an activation, and projected back down. That is why pages like ReLU, Leaky ReLU, SiLU, and SwiGLU matter: they are different answers to the same question of how the hidden layer should shape its intermediate values. The broad activation idea also shows up in other architectures, but the feed-forward slot in transformers is the most common place to study it first.Common Confusions
Hidden activations are not the same as output softmax. Hidden activations are internal signals passed between layers after nonlinear shaping; softmax is a final step on the output head that converts logits into a probability distribution over vocabulary items or classes. People also use the word activation in two nearby ways: for the activation function itself and for the resulting hidden values. Both usages are common, but neither is the same as the softmax probabilities you read at the end of a classifier or language model.References
- Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016, https://www.deeplearningbook.org/.
- Nair, Vinod, and Geoffrey E. Hinton. "Rectified Linear Units Improve Restricted Boltzmann Machines." ICML, 2010, https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf.
- Elfwing, Stefan, Eiji Uchibe, and Kenji Doya. "Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning." arXiv, 2017, https://arxiv.org/abs/1702.03118.