Modules

Absolute positional embeddings give each token position its own index-specific vector, then add that vector directly to the token embedding before attention runs.

Attention with linear biases adds distance-based penalties to attention logits so models can extrapolate to longer sequences without explicit position embeddings.

How transformer blocks mix information across token positions using query, key, and value projections.

Batch-level normalization that rescales activations using statistics pooled across examples, common in convolutional networks but uncommon inside modern transformers.

An attention pattern that lets each token read context on both sides when the model is allowed to see the full sequence.

An attention variant that divides the attention map into token blocks and computes scores only for selected block regions instead of every token pair.

A subword tokenizer that learns frequent text pieces by repeatedly merging common neighboring symbols.

A tokenizer design that starts from bytes so models can represent arbitrary text without unknown-character gaps.

An attention pattern that lets each token read earlier tokens and itself, but not later tokens, so the model can predict text one step at a time.

A vision tokenizer that turns fixed-size image patches into embedding vectors for transformer-style image encoders.

A DeepSeek-V4 attention variant that mixes compression and sparse access so very long prompts stay tractable.

An attention pattern where queries come from one stream while keys and values come from a different memory source.

The MoE block style emphasized in DeepSeek-V4, where expert routing is a central architecture and serving choice rather than a small add-on.

A transformer-style block that updates noisy image or latent patch tokens during diffusion denoising, steered by timestep and optional conditioning signals.

A neural network whose signals move from input to output through weighted layers without recurrent loops back into earlier layers.

A sequence-mixing module that updates a compact recurrent memory state with input-dependent gating and delta-rule writes instead of materializing a full attention matrix.

A smooth activation function that keeps small negative values and is common in transformer feed-forward blocks.

Channel-group normalization that rescales each example using per-group statistics, often used when batch norm is awkward or unstable.

An attention variant that reduces key-value cache memory by sharing key-value heads across query groups.

A DeepSeek-V4 long-context attention path that relies on a more aggressively compressed memory view when full-detail history is too costly.

A smooth activation function that maps each value into a centered -1-to-1 range.

Per-token mean-and-variance normalization that rescales each hidden vector before the next sublayer in a transformer block.

An activation function that keeps a small negative slope instead of turning every negative value into zero.

Learned positional embeddings train a position table alongside token embeddings, so each index gets a position vector the model can adapt during training.

An attention variant that replaces explicit softmax dot-product attention with kernel or feature-map formulations that scale near-linearly with sequence length.

A broad attention pattern that lets each query look only at a nearby neighborhood instead of the full sequence.

LongRoPE is a RoPE extension method that rescales rotary dimensions and positions non-uniformly so a model can stretch much farther into long-context ranges.

A transformer architecture that applies one shared block repeatedly over many loop iterations instead of stacking many distinct layers.

A sequence-mixing module that updates a compact recurrent state as each token arrives, using input-dependent parameters to decide what the state stores, forgets, and emits.

A DeepSeek-V4 block-level wiring change that gives layers a richer shared path without making every update fully dense.

A sparse feed-forward layer that routes each token to a small subset of expert layers instead of one dense shared block.

The baseline attention design that gives every query head its own key-value head pair.

An attention variant that compresses key-value cache storage into a low-rank latent space while keeping distinct query heads.

An attention variant that shares one key-value head across all query heads to minimize key-value cache memory.

A training objective that asks each position to predict several future tokens through independent output heads on a shared model trunk.

NoPE means a model removes explicit positional encoding and tests how much order signal attention and training can recover without a dedicated position mechanism.

NTK-aware RoPE scaling changes the rotary frequency base so a RoPE model keeps more useful relative behavior when it is asked to run beyond its trained context length.

Positional interpolation extends a RoPE model by compressing new token positions back into the position range the model was originally trained to handle.

Query-key normalization that rescales attention queries and keys before their dot products, changing the attention score path rather than the whole residual stream.

A simple activation function that keeps positive values and turns negative values into zero.

Relative position bias changes attention scores with distance-aware bias terms so the model reasons about how far apart tokens are instead of storing a fixed vector for every position.

Per-token root-mean-square scaling that rescales each hidden vector without mean centering, as used in many modern decoder transformers.

Rotary position embedding rotates query and key vectors by token index so attention scores depend on relative distance, as used in Llama-class decoder models.

A tokenizer system that learns subword pieces directly from raw text, including spaces, without requiring word splitting first.

A smooth activation function that maps each value into a 0-to-1 gate.

A smooth activation function that scales each value by its own sigmoid score.

Sinusoidal positional embeddings compute fixed wave-shaped position vectors from token index, so models get absolute order without learning a separate parameter row for each position.

An attention variant that restricts each query to a fixed local window of key positions instead of the full sequence.

An attention variant that computes scores only for a subset of token pairs instead of every query-key combination.

The default dense feed-forward layer that expands, activates, and projects each token state after attention.

SuperHOT RoPE stretches rotary positions across a longer range, then fine-tunes the model so the compressed position map still behaves well at the new window.

A gated feed-forward design that multiplies a value branch by a swish gate before projecting back.

T5 relative position bias buckets token distances into ranges, then adds a learned bias per bucket so attention can reason about near and far positions without a full absolute table.

A failure mode where text is split, labeled, or wrapped differently from what the model was trained or served to expect.

A U-shaped convolutional network that denoises images by mixing multiscale context down a downsampling path, through a bottleneck, and back up an upsampling path with skip connections.

A subword tokenizer that keeps a candidate vocabulary and picks the highest-scoring whole segmentation instead of replaying merge rules.

A subword tokenizer that grows a vocabulary of reusable pieces and usually tokenizes each word with the longest matching subword sequence.

YaRN is a RoPE extension method that mixes interpolation-style scaling with continued training so a model can use a larger context window more efficiently.