Feed-forward

A larger open-weights Mixtral sparse mixture-of-experts model with 141 billion total parameters, about 39 billion active per token, and a 64,000-token context window under Apache 2.0.

The first open-weights Mixtral sparse mixture-of-experts model with 46.7 billion total parameters, about 12.9 billion active per token, and a 32,768-token context window under Apache 2.0.

The sparse-expert Qwen 3.6 open-weights model with 35 billion total parameters, about 3 billion active per token, multimodal inputs, and a 262,144-token native context window.

An encoder-decoder transformer from Google Research that reformulates every NLP task as reading text and writing text in a shared token space.

The MoE block style emphasized in DeepSeek-V4, where expert routing is a central architecture and serving choice rather than a small add-on.

A neural network whose signals move from input to output through weighted layers without recurrent loops back into earlier layers.

A sparse feed-forward layer that routes each token to a small subset of expert layers instead of one dense shared block.

The default dense feed-forward layer that expands, activates, and projects each token state after attention.

A gated feed-forward design that multiplies a value branch by a swish gate before projecting back.

A broad sparse-routing architecture pattern that activates only a few expert sub-networks per token instead of one dense path for every token.

A serving system that overlaps expert routing, communication, and compute so sparse MoE execution does not stall on movement alone.