Swish Gated Linear Unit
A gated feed-forward design that multiplies a value branch by a swish gate before projecting back.
The swish gated linear unit (SwiGLU) splits a feed-forward layer into value and gate branches, applies the sigmoid linear unit (SiLU) to the gate, and multiplies the two branches so the layer can filter features more selectively.
At a glance
Optimizes
- Feed Forward Expressiveness
What It Is
The swish gated linear unit (SwiGLU) is a gated feed-forward design used in many modern language models. Instead of using one expanded hidden branch, the layer creates a value branch and a gate branch. The gate branch passes through SiLU, then scales the value branch element by element before the layer projects back to the model width.Why It Exists
SwiGLU optimizes for a more selective dense hidden path. Instead of treating every hidden feature the same way, it lets a learned gate strengthen or suppress each feature before the final projection.How It Works
The input state enters two learned projections. One branch carries candidate values. The other branch becomes a SiLU gate. The two branches multiply elementwise, and the layer then projects the result back to the model width. The path stays dense because every token still uses the same shared block.Token state h_tValue projection W_vValue branchElementwise multiplyProject back with W_oUpdated token stateGate projection W_gSiLUSwiGLUToken state h_t to Value projection W_vToken state h_t to Gate projection W_gValue projection W_v to Value branchGate projection W_g to SiLUValue branch to Elementwise multiplySiLU to Elementwise multiplyElementwise multiply to Project back with W_oProject back with W_o to Updated token state
Math Or Compute Schema
The dense baseline below shows the simpler one-branch feed-forward layer. The SwiGLU formula shows the extra gate that turns that same slot into a two-branch gated path.- Hidden state for token position t.
- Projection into the wide hidden layer.
- Projection back to model width.
- Pointwise activation inside the dense hidden layer.
- Hidden state for token position t.
- Projection that builds the value branch.
- Projection that builds the gate branch.
- Projection back to model width after gating.
- Elementwise multiplication between the value and gate branches.
Compared To Nearby Modules
SwiGLU stays closer to the standard dense feed-forward baseline than mixture of experts does. It changes how features are filtered inside one dense block rather than turning the slot into a sparse routed expert system.| Comparison dimension | SwiGLU | Standard feed-forward | Mixture of experts |
|---|---|---|---|
| Path shape | Value branch multiplied by a SiLU gate branch | One dense expand -> activate -> project path | Router selects a few expert MLPs, then merges them |
| Active compute per token | All hidden units plus a learned gate | All hidden units in one shared feed-forward block | Only top-k experts activate for each token |
| Main tradeoff | More selective dense features, but still full dense compute | Simpler path, but less selective hidden filtering | More capacity, but routing and sparse training become part of the design |