Feed-Forward Network
A neural network whose signals move from input to output through weighted layers without recurrent loops back into earlier layers.
A feed-forward network is a neural network that pushes an input through one or more learned weight matrices and nonlinear steps until it produces an output, without sending the signal backward through cycles during the forward pass.
At a glance
Released
March 1989
Authors
Kurt Hornik, Maxwell Stinchcombe, Halbert White
Optimizes
- Per Token Capacity
What It Is
A feed-forward network, often shortened to FFN, is a neural network whose computation flows from input to output in one direction. Each layer multiplies the current values by learned weights, usually adds a bias, applies a nonlinear activation, and passes the result onward. The key idea is general: no recurrent loop is needed in the forward path.Why It Exists
Feed-forward networks exist because a stack of learned weighted transformations can turn raw inputs into more useful internal features and final outputs. Coupled with an activation function, the FFN can express non linear functions and representations. As a consequence the FFN is able to represent complex concepts in real life.How It Works
An input vector enters the first learned projection, gets transformed into a hidden representation, passes through an activation, and continues toward the output. In a simple dense feed-forward network, every example uses the same hidden layers.Token state h_tExpand to hidden widthOne shared hidden layerActivation on that hidden layerProject backUpdated token stateTake data, multiply by matrix, pass through activation, returnffn-internalsFeed forward networkToken state h_t to Expand to hidden widthExpand to hidden width to One shared hidden layerOne shared hidden layer to Activation on that hidden layerActivation on that hidden layer to Project backProject back to Updated token state
Math Or Compute Schema
This formula shows the standard feed-forward pattern: project the input into hidden computation, apply a nonlinear transformation, then map the result onward.- Input vector.
- Projection into the hidden layer.
- Projection from the hidden layer toward the output.
- Pointwise activation applied in the hidden computation.
Example Architectures
Feed-forward networks appear across classic multilayer perceptrons, convolutional classifiers, recommendation models, and transformer blocks. In transformers they usually appear as the position-wise dense or sparse sublayer after attention, but that is only one important use case.Limitations And Tradeoffs
A feed-forward network only applies the transformations its architecture makes available. A plain dense version can be simple and stable, but expensive when hidden layers get large. Gated and sparse variants can be more expressive or more parameter-efficient, but they add routing, balancing, or optimization complexity.Why It Still Matters
Feed-forward networks remain one of the core building blocks of modern machine learning. Even when a paper introduces a new gated block or sparse expert system, it is often best understood as a new way to organize feed-forward computation rather than as a completely separate idea.References
- Hornik, Kurt, Maxwell Stinchcombe, and Halbert White. "Multilayer Feedforward Networks are Universal Approximators." Neural Networks, vol. 2, no. 5, 1989, pp. 359-366. https://doi.org/10.1016/0893-6080(89)90020-8.
- Vaswani, Ashish, et al. "Attention Is All You Need." arXiv, 2017, https://arxiv.org/abs/1706.03762.