Standard Feed-Forward Network
The default dense feed-forward layer that expands, activates, and projects each token state after attention.
A standard feed-forward network, often shortened to standard FFN, is the default dense layer after attention in a transformer block, using one shared expand-activate-project path for every token.
At a glance
Released
June 2017
Authors
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.
Optimizes
- Per Token Capacity
What It Is
A standard feed-forward network, often shortened to standard FFN, is the default dense feed-forward block used in most transformer layers. After attention updates a token state, the layer sends that state through one wide linear layer, applies a pointwise activation, and projects it back to the model width.Why It Exists
The standard feed-forward network optimizes for a simple, uniform dense path after attention. Every token uses the same parameters and the same full hidden width, which keeps the block easy to reason about and easy to compare against later variants.How It Works
The token state enters one shared dense block, expands into a wider hidden width, passes through an activation, and then projects back to the model width. No router chooses between experts, and no separate gate branch decides which hidden features stay active.Token state h_tExpand to hidden widthOne shared hidden layerActivation on that hidden layerProject backUpdated token stateEvery token uses one shared expand -> hidden layer -> activate -> project pathToken state h_t to Expand to hidden widthExpand to hidden width to One shared hidden layerOne shared hidden layer to Activation on that hidden layerActivation on that hidden layer to Project backProject back to Updated token state
Math Or Compute Schema
The dense feed-forward formula below is the baseline for the whole family. It shows the single shared expand, activate, and project path that many later variants start from.- Hidden state for token position t.
- Projection into the wide hidden layer.
- Projection back to model width.
- Pointwise activation inside the hidden layer.
Compared To Nearby Modules
Compared with other feed-forward modules, the standard feed-forward network keeps the most direct dense path. The main tradeoff is that every token activates the full hidden block every time.| Comparison dimension | Standard feed-forward | SwiGLU | Mixture of experts |
|---|---|---|---|
| Path shape | One dense expand -> activate -> project path | Value branch multiplied by a SiLU gate branch | Router chooses a few expert layers, then merges them |
| Active compute per token | All hidden units in one shared feed-forward block | All hidden units plus a learned gate | Only top-k experts activate for each token |
| Main tradeoff | Simple and predictable, but every token pays for the full block | More expressive, but the block is less minimal than the dense baseline | Much more capacity, but routing and load balancing add complexity |