Attention

A lighter DeepSeek-V4 serving profile that keeps the same conceptual stack while prioritizing faster deployment paths.

The larger DeepSeek-V4 reasoning model, built around long-context attention, MoE routing, and coupled post-training and serving choices.

A large open-weights GLM model that pairs sparse-attention and mixture-of-experts routing with an agentic-engineering training story.

A later GLM line member that extends sparse-attention mixture-of-experts architecture with a solid 1M-token context and long-horizon coding emphasis.

A large decoder-only language model that made in-context prompting and few-shot text generation a mainstream reference point.

Meta's open-weight Llama-family decoder-only transformer release, spanning dense and instruction-tuned checkpoints with long-context support.

A larger open-weights Mixtral sparse mixture-of-experts model with 141 billion total parameters, about 39 billion active per token, and a 64,000-token context window under Apache 2.0.

The first open-weights Mixtral sparse mixture-of-experts model with 46.7 billion total parameters, about 12.9 billion active per token, and a 32,768-token context window under Apache 2.0.

The smallest dense Qwen3 open-weight checkpoint: a 0.6-billion-parameter causal language model with 32,768-token context, Apache 2.0 licensing, and post-training for reasoning and instruction following.

The smallest published Qwen3.5 open-weights checkpoint with multimodal inputs, a hybrid decoder stack, and a 262,144-token native context window.

The dense Qwen 3.6 open-weights model with a 27-billion-parameter stack, multimodal inputs, and a 262,144-token native context window.

The sparse-expert Qwen 3.6 open-weights model with 35 billion total parameters, about 3 billion active per token, multimodal inputs, and a 262,144-token native context window.

How transformer blocks mix information across token positions using query, key, and value projections.

An attention pattern that lets each token read context on both sides when the model is allowed to see the full sequence.

An attention variant that divides the attention map into token blocks and computes scores only for selected block regions instead of every token pair.

An attention pattern that lets each token read earlier tokens and itself, but not later tokens, so the model can predict text one step at a time.

A DeepSeek-V4 attention variant that mixes compression and sparse access so very long prompts stay tractable.

An attention pattern where queries come from one stream while keys and values come from a different memory source.

A sequence-mixing module that updates a compact recurrent memory state with input-dependent gating and delta-rule writes instead of materializing a full attention matrix.

An attention variant that reduces key-value cache memory by sharing key-value heads across query groups.

A DeepSeek-V4 long-context attention path that relies on a more aggressively compressed memory view when full-detail history is too costly.

An attention variant that replaces explicit softmax dot-product attention with kernel or feature-map formulations that scale near-linearly with sequence length.

A broad attention pattern that lets each query look only at a nearby neighborhood instead of the full sequence.

A DeepSeek-V4 block-level wiring change that gives layers a richer shared path without making every update fully dense.

The baseline attention design that gives every query head its own key-value head pair.

An attention variant that compresses key-value cache storage into a low-rank latent space while keeping distinct query heads.

An attention variant that shares one key-value head across all query heads to minimize key-value cache memory.

An attention variant that restricts each query to a fixed local window of key positions instead of the full sequence.

An attention variant that computes scores only for a subset of token pairs instead of every query-key combination.

Saved key and value tensors from earlier tokens that let later generation steps reuse attention state instead of recomputing the full prefix.

The prompt-processing stage that reads the full input once, builds the first attention state, and sets up generation before any reply token appears.

The serving setup that separates prompt processing from token-by-token generation because the two stages stress hardware, memory, and latency in different ways.

How a token reads other tokens from the same sequence so a transformer can mix context before producing the next representation.

The 2017 paper that introduced the transformer, using self-attention as the main way tokens mix information in encoder-decoder sequence models.

The paper that introduced deep bidirectional transformer encoders trained with masked language modeling and fine-tuned for language understanding tasks.

A technical report that bundles DeepSeek-V4 architecture, long-context attention, post-training methods, and serving systems into one connected release.

A generation paradigm that produces outputs one discrete step at a time, each step conditioned on everything generated so far.

The repeated next-token stage that reuses the KV cache and turns inter-token latency into the main reader experience after prefill finishes.

The saved key and value tensors a decoder keeps from earlier tokens so later tokens can reuse attention state instead of recomputing the full prefix.

The smallest unit of text a language model reads and predicts—usually a word piece, not always a whole word. Each token ID maps to a dense vector through vector embedding of model hidden size before attention runs.