Manifold-Constrained Hyper-Connections
A DeepSeek-V4 block-level wiring change that gives layers a richer shared path without making every update fully dense.
Deep transformer stacks often lose useful intermediate state as they get deeper. Manifold-constrained hyper-connections try to preserve richer cross-layer exchange while keeping the update path structured enough to remain trainable.
At a glance
Optimizes
- Depth Stability
- Information Routing
Example models
What It Is
This module is a block-level connection pattern rather than a new attention rule by itself. The paper presents it as a way for the residual pathway to carry more structured information between layers.Why It Exists
The goal is depth stability and better reuse of useful state, especially when the rest of the stack is already complicated by sparse experts and long-context attention.How It Works
The incoming residual stream is projected into a constrained shared space, transformed, then merged back into the block update. The exact implementation details matter less here than the high-level fact that the residual route is no longer treated as one simple passthrough.Residual stateConstrained shared routeMerged block updateResidual state to Constrained shared routeConstrained shared route to Merged block update
Math Or Compute Schema
This formula is only a teaching sketch. It highlights the idea that a residual state is mapped into a constrained latent route before it is merged back.- the incoming hidden or residual state
- the constrained shared representation
- the mapping into the constrained route
- the update merged back into the block state