Context Window

A lighter DeepSeek-V4 serving profile that keeps the same conceptual stack while prioritizing faster deployment paths.

The larger DeepSeek-V4 reasoning model, built around long-context attention, MoE routing, and coupled post-training and serving choices.

Open model family from Google DeepMind with Gemma 4 as the current main line, spanning Effective 2B and 4B edge checkpoints through 12B, 26B mixture-of-experts, and 31B dense variants with native text, image, and audio input, thinking-mode reasoning, and Apache 2.0 licensing.

A large open-weights GLM model that pairs sparse-attention and mixture-of-experts routing with an agentic-engineering training story.

A later GLM line member that extends sparse-attention mixture-of-experts architecture with a solid 1M-token context and long-horizon coding emphasis.

A large decoder-only language model that made in-context prompting and few-shot text generation a mainstream reference point.

Meta's open-weight Llama-family decoder-only transformer release, spanning dense and instruction-tuned checkpoints with long-context support.

A larger open-weights Mixtral sparse mixture-of-experts model with 141 billion total parameters, about 39 billion active per token, and a 64,000-token context window under Apache 2.0.

The first open-weights Mixtral sparse mixture-of-experts model with 46.7 billion total parameters, about 12.9 billion active per token, and a 32,768-token context window under Apache 2.0.

Open-weights 120B-total / 12B-active hybrid Mamba-Attention Mixture-of-Experts text model from NVIDIA's Nemotron 3 family, with long-context support up to 1M tokens and NVFP4/BF16 deployment variants.

The smallest dense Qwen3 open-weight checkpoint: a 0.6-billion-parameter causal language model with 32,768-token context, Apache 2.0 licensing, and post-training for reasoning and instruction following.

The smallest published Qwen3.5 open-weights checkpoint with multimodal inputs, a hybrid decoder stack, and a 262,144-token native context window.

The dense Qwen 3.6 open-weights model with a 27-billion-parameter stack, multimodal inputs, and a 262,144-token native context window.

The sparse-expert Qwen 3.6 open-weights model with 35 billion total parameters, about 3 billion active per token, multimodal inputs, and a 262,144-token native context window.

An attention variant that divides the attention map into token blocks and computes scores only for selected block regions instead of every token pair.

A DeepSeek-V4 attention variant that mixes compression and sparse access so very long prompts stay tractable.

A sequence-mixing module that updates a compact recurrent memory state with input-dependent gating and delta-rule writes instead of materializing a full attention matrix.

A DeepSeek-V4 long-context attention path that relies on a more aggressively compressed memory view when full-detail history is too costly.

A broad attention pattern that lets each query look only at a nearby neighborhood instead of the full sequence.

LongRoPE is a RoPE extension method that rescales rotary dimensions and positions non-uniformly so a model can stretch much farther into long-context ranges.

A sequence-mixing module that updates a compact recurrent state as each token arrives, using input-dependent parameters to decide what the state stores, forgets, and emits.

NTK-aware RoPE scaling changes the rotary frequency base so a RoPE model keeps more useful relative behavior when it is asked to run beyond its trained context length.

Positional interpolation extends a RoPE model by compressing new token positions back into the position range the model was originally trained to handle.

An attention variant that restricts each query to a fixed local window of key positions instead of the full sequence.

SuperHOT RoPE stretches rotary positions across a longer range, then fine-tunes the model so the compressed position map still behaves well at the new window.

YaRN is a RoPE extension method that mixes interpolation-style scaling with continued training so a model can use a larger context window more efficiently.

Common techniques that stretch usable context beyond a model's original training window without rebuilding the entire architecture from scratch.

Why stretching sequence length strains attention compute, KV-cache memory, position schemes, and retrieval quality even when a model advertises a large context window.

A technical report that bundles DeepSeek-V4 architecture, long-context attention, post-training methods, and serving systems into one connected release.

A serving system concern that covers the live state the runtime must keep or move while answering requests, including model weights, KV cache growth, allocator overhead, and memory bandwidth limits.

A serving system that spills part of the KV cache to disk so very long sessions remain workable without keeping every token state in expensive fast memory.

The maximum number of tokens a model can attend over in one forward pass, distinct from training sequence length and how many tokens it may emit in a session.