Foundations

An encoder-only transformer that learns bidirectional text representations through masked language modeling pretraining.

A dual-encoder multimodal model that learns to align text and images in one shared embedding space.

Open omnimodal world model family from NVIDIA for physical AI, combining autoregressive reasoning with diffusion-based generation across text, image, video, audio, and action outputs through Nano and Super checkpoints.

A lighter DeepSeek-V4 serving profile that keeps the same conceptual stack while prioritizing faster deployment paths.

The larger DeepSeek-V4 reasoning model, built around long-context attention, MoE routing, and coupled post-training and serving choices.

A Black Forest Labs image-generation model family that turns text prompts into images using rectified-flow transformer backbones in latent space.

Open model family from Google DeepMind with Gemma 4 as the current main line, spanning Effective 2B and 4B edge checkpoints through 12B, 26B mixture-of-experts, and 31B dense variants with native text, image, and audio input, thinking-mode reasoning, and Apache 2.0 licensing.

A large open-weights GLM model that pairs sparse-attention and mixture-of-experts routing with an agentic-engineering training story.

A later GLM line member that extends sparse-attention mixture-of-experts architecture with a solid 1M-token context and long-horizon coding emphasis.

A large decoder-only language model that made in-context prompting and few-shot text generation a mainstream reference point.

Meta's open-weight Llama-family decoder-only transformer release, spanning dense and instruction-tuned checkpoints with long-context support.

An open-weight diffusion-transformer model that generates synchronized video and audio from text, image, and audio conditioning.

A larger open-weights Mixtral sparse mixture-of-experts model with 141 billion total parameters, about 39 billion active per token, and a 64,000-token context window under Apache 2.0.

The first open-weights Mixtral sparse mixture-of-experts model with 46.7 billion total parameters, about 12.9 billion active per token, and a 32,768-token context window under Apache 2.0.

Open-weights 120B-total / 12B-active hybrid Mamba-Attention Mixture-of-Experts text model from NVIDIA's Nemotron 3 family, with long-context support up to 1M tokens and NVFP4/BF16 deployment variants.

The smallest dense Qwen3 open-weight checkpoint: a 0.6-billion-parameter causal language model with 32,768-token context, Apache 2.0 licensing, and post-training for reasoning and instruction following.

The smallest published Qwen3.5 open-weights checkpoint with multimodal inputs, a hybrid decoder stack, and a 262,144-token native context window.

The dense Qwen 3.6 open-weights model with a 27-billion-parameter stack, multimodal inputs, and a 262,144-token native context window.

The sparse-expert Qwen 3.6 open-weights model with 35 billion total parameters, about 3 billion active per token, multimodal inputs, and a 262,144-token native context window.

An encoder-decoder transformer from Google Research that reformulates every NLP task as reading text and writing text in a shared token space.

Absolute positional embeddings give each token position its own index-specific vector, then add that vector directly to the token embedding before attention runs.

Attention with linear biases adds distance-based penalties to attention logits so models can extrapolate to longer sequences without explicit position embeddings.

Batch-level normalization that rescales activations using statistics pooled across examples, common in convolutional networks but uncommon inside modern transformers.

A subword tokenizer that learns frequent text pieces by repeatedly merging common neighboring symbols.

A vision tokenizer that turns fixed-size image patches into embedding vectors for transformer-style image encoders.

The MoE block style emphasized in DeepSeek-V4, where expert routing is a central architecture and serving choice rather than a small add-on.

A transformer-style block that updates noisy image or latent patch tokens during diffusion denoising, steered by timestep and optional conditioning signals.

A neural network whose signals move from input to output through weighted layers without recurrent loops back into earlier layers.

A smooth activation function that keeps small negative values and is common in transformer feed-forward blocks.

Channel-group normalization that rescales each example using per-group statistics, often used when batch norm is awkward or unstable.

A smooth activation function that maps each value into a centered -1-to-1 range.

Per-token mean-and-variance normalization that rescales each hidden vector before the next sublayer in a transformer block.

An activation function that keeps a small negative slope instead of turning every negative value into zero.

Learned positional embeddings train a position table alongside token embeddings, so each index gets a position vector the model can adapt during training.

LongRoPE is a RoPE extension method that rescales rotary dimensions and positions non-uniformly so a model can stretch much farther into long-context ranges.

A transformer architecture that applies one shared block repeatedly over many loop iterations instead of stacking many distinct layers.

A DeepSeek-V4 block-level wiring change that gives layers a richer shared path without making every update fully dense.

A sparse feed-forward layer that routes each token to a small subset of expert layers instead of one dense shared block.

A training objective that asks each position to predict several future tokens through independent output heads on a shared model trunk.

NoPE means a model removes explicit positional encoding and tests how much order signal attention and training can recover without a dedicated position mechanism.

NTK-aware RoPE scaling changes the rotary frequency base so a RoPE model keeps more useful relative behavior when it is asked to run beyond its trained context length.

Positional interpolation extends a RoPE model by compressing new token positions back into the position range the model was originally trained to handle.

Query-key normalization that rescales attention queries and keys before their dot products, changing the attention score path rather than the whole residual stream.

A simple activation function that keeps positive values and turns negative values into zero.

Relative position bias changes attention scores with distance-aware bias terms so the model reasons about how far apart tokens are instead of storing a fixed vector for every position.

Per-token root-mean-square scaling that rescales each hidden vector without mean centering, as used in many modern decoder transformers.

Rotary position embedding rotates query and key vectors by token index so attention scores depend on relative distance, as used in Llama-class decoder models.

A tokenizer system that learns subword pieces directly from raw text, including spaces, without requiring word splitting first.

A smooth activation function that maps each value into a 0-to-1 gate.

A smooth activation function that scales each value by its own sigmoid score.

Sinusoidal positional embeddings compute fixed wave-shaped position vectors from token index, so models get absolute order without learning a separate parameter row for each position.

The default dense feed-forward layer that expands, activates, and projects each token state after attention.

SuperHOT RoPE stretches rotary positions across a longer range, then fine-tunes the model so the compressed position map still behaves well at the new window.

A gated feed-forward design that multiplies a value branch by a swish gate before projecting back.

T5 relative position bias buckets token distances into ranges, then adds a learned bias per bucket so attention can reason about near and far positions without a full absolute table.

A failure mode where text is split, labeled, or wrapped differently from what the model was trained or served to expect.

A U-shaped convolutional network that denoises images by mixing multiscale context down a downsampling path, through a bottleneck, and back up an upsampling path with skip connections.

A subword tokenizer that keeps a candidate vocabulary and picks the highest-scoring whole segmentation instead of replaying merge rules.

A subword tokenizer that grows a vocabulary of reusable pieces and usually tokenizes each word with the longest matching subword sequence.

YaRN is a RoPE extension method that mixes interpolation-style scaling with continued training so a model can use a larger context window more efficiently.

The nonlinear step that reshapes a layer's output so stacked layers can model richer patterns than repeated linear transforms alone.

The post-pretraining goal of shaping a broadly capable base model toward helpful, safer, preference-following behavior before or during deployment.

A positional idea that subtracts larger attention penalties for farther token pairs instead of storing explicit position embeddings.

A diffusion inference technique that steers denoising by comparing prompt-conditioned and prompt-free predictions, then blending them with a guidance scale.

Common techniques that stretch usable context beyond a model's original training window without rebuilding the entire architecture from scratch.

A learned vector that turns discrete tokens, image patches, or other inputs into numbers a model can process.

Floating-point operations counted to estimate how much arithmetic work a model needs during inference, before teams compare serving cost, throughput plans, or hardware capability.

A generative training objective that teaches a model how samples should move from a simple or noisy starting state toward real data.

Saved key and value tensors from earlier tokens that let later generation steps reuse attention state instead of recomputing the full prefix.

A learned compressed representation space where models store internal codes instead of raw pixels or token ids, then decode or denoise back to visible outputs.

How quickly weights, activations, and KV cache bytes can move between memory and compute during model serving, and why that movement can cap useful throughput.

A broad sparse-routing architecture pattern that activates only a few expert sub-networks per token instead of one dense path for every token.

Keeping activations on a predictable scale so deep model stacks stay trainable and their sublayers receive stable inputs.

Learning from examples or trajectories produced by the model or agent policy that is currently being updated.

A small generated concept page that demonstrates the page-spec bundle workflow end to end.

How transformers inject order into otherwise permutation-invariant attention through absolute, relative, and rotary position schemes.

The prompt-processing stage that reads the full input once, builds the first attention state, and sets up generation before any reply token appears.

The serving setup that separates prompt processing from token-by-token generation because the two stages stress hardware, memory, and latency in different ways.

Training-time and model-design choices that discourage brittle memorization so a model is more likely to perform well on new examples.

A throughput-ceiling chart that shows when memory movement or arithmetic compute limits attainable performance, using arithmetic intensity to explain which bound is active.

How a token reads other tokens from the same sequence so a transformer can mix context before producing the next representation.

Training material produced or transformed by one model to teach another, distinct from raw web corpora collected for pretraining or demonstrations written directly by people.

A positive scaling factor applied to logits before softmax to make the next-token distribution sharper or flatter.

Extra prompt-derived vectors that steer an image generator while the denoiser removes noise step by step.

How tokenizers turn raw text into reusable pieces before embeddings, attention, and next-token prediction begin.

How attention, feed-forward layers, normalization, residuals, and position information repeat inside each decoder-style transformer block.

Models that produce moving visual sequences by generating or refining frames—or visual tokens arranged over time—while keeping motion, identity, and appearance coherent from one moment to the next.

How images and video frames become patches, discrete codes, or latent tokens so sequence-based models can read visual data.

Why stretching sequence length strains attention compute, KV-cache memory, position schemes, and retrieval quality even when a model advertises a large context window.

The 2017 paper that introduced the transformer, using self-attention as the main way tokens mix information in encoder-decoder sequence models.

The paper that introduced deep bidirectional transformer encoders trained with masked language modeling and fine-tuned for language understanding tasks.

A technical report that bundles DeepSeek-V4 architecture, long-context attention, post-training methods, and serving systems into one connected release.

A technical report that introduced GPT-2 as a large decoder-only language model trained with broad next-token pretraining and byte-level BPE tokenization.

The research paper behind Stable Diffusion-style image generation: compress images into a latent space, denoise there instead of in pixel space, then decode the result.

The CLIP paper introduced contrastive language-image pretraining with separate image and text encoders that map paired captions and images into one shared embedding space.

July 2026

A narrative arc from denoising diffusion probabilistic models and convolutional U-Net backbones through CLIP-conditioned image systems, latent diffusion, diffusion transformers, flow matching, and source-backed open video and world-model examples.

July 2026

Why modern language-model quality still starts with internet-scale pretraining but increasingly depends on mid-training, post-training, preference feedback, verifiable rewards, and distillation loops.

The pretraining-style regime where a denoiser learns to predict noise or a clean-direction signal from corrupted examples across a noise schedule.

A preference-optimization training regime that teaches a model from preferred versus rejected answer pairs without fitting a separate reward model and PPO loop first.

A teacher-student training regime where a smaller model learns from supervision produced by a larger or stronger teacher model.

A training-time regularization method that randomly disables a subset of activations during learning and uses the full network, with appropriate scaling, at inference.

A DeepSeek-V4 training step that prepares the model for very low precision by exposing it to quantization effects during training.

A groupwise alignment training regime that samples multiple candidate answers for the same prompt and updates the model from how those answers rank relative to one another.

A supervised post-training regime that teaches a model to follow user instructions by training on prompts paired with desired answers.

Continued training on a pretrained base checkpoint before final post-training, using targeted data or objectives to extend capability without yet shaping final assistant behavior.

A DeepSeek-V4 post-training routine where the student is refined on trajectories produced under its current policy rather than only on frozen offline traces.

The training stages after base pretraining that reshape how a language model behaves in use, including instruction following, preference alignment, and safety shaping before deployment.

The large base-model training stage where a transformer learns broad language patterns from massive token sequences before later alignment or deployment shaping.

A post-training regime that updates a model with clipped reinforcement-learning steps so RLHF behavior improves without each update moving too far at once.

A post-training alignment workflow that collects human preference signals, fits a reward model, and optimizes the language-model policy toward preferred behavior.

A post-training reinforcement learning regime where rewards come from externally checkable task outcomes rather than human preference labels alone.

A DeepSeek-V4 post-training pattern that sharpens separate capability areas before they are merged back into the final stack.

A post-training regime that adapts a capable base model by training on labeled input-output examples.

A serving system that groups multiple requests or decode steps together so accelerators stay busier, raising throughput while often adding wait time for each individual request.

A serving system that keeps an active decode set refilled as requests finish or become compatible, so accelerators spend less time waiting for a whole batch to turn over.

A serving system practice that turns a trained model plus its runtime stack into a live, versioned inference service on real hardware.

A serving system that waits briefly for compatible queued requests, then runs them together so accelerators stay busier while trading a small queueing delay for higher throughput.

A serving system that overlaps expert routing, communication, and compute so sparse MoE execution does not stall on movement alone.

The serving runtime layer that takes model weights plus live requests and turns them into token outputs by coordinating kernels, scheduling, memory movement, and cache state on real hardware.

A serving system concern that covers the live state the runtime must keep or move while answering requests, including model weights, KV cache growth, allocator overhead, and memory bandwidth limits.

A serving system that spills part of the KV cache to disk so very long sessions remain workable without keeping every token state in expensive fast memory.

A serving system that decides which queued request work runs next so operators can balance latency, fairness, throughput, and memory pressure.

A serving system that decides which model, hardware tier, or execution path should handle a request so operators can balance cost, latency, safety, and specialization.

A serving technique that uses a cheaper draft path to propose several next tokens, then asks the main model to verify them so decode-heavy requests can finish with fewer expensive turns.

The numeric output of a layer after its linear transform and nonlinearity, passed forward through the network.

Post-training steps that steer a model toward helpful, honest, and policy-compliant behavior—often via preference data, reward models, or safety filters.

The blueprint that defines how a model's layers, modules, and data flow connect during inference.

A generation paradigm that produces outputs one discrete step at a time, each step conditioned on everything generated so far.

The backward pass that applies the chain rule on a computational graph to compute gradients for every differentiable parameter.

A finer-grained part inside a module, such as an attention head or projection layer.

A record of which operations produced each tensor so automatic differentiation can run backward.

Extra inputs—text prompts, class labels, images, or guidance weights—that steer a generative model without retraining the full parameter stack each time.

The maximum number of tokens a model can attend over in one forward pass, distinct from training sequence length and how many tokens it may emit in a session.

The repeated next-token stage that reuses the KV cache and turns inter-token latency into the main reader experience after prefill finishes.

A network stack that turns internal representations into outputs—tokens, pixels, or structured predictions—often one step at a time with attention to prior context.

A generation paradigm that starts from noisy latents and iteratively removes noise—or adds structure—until a clean sample emerges.

A model trained to score, rank, or classify inputs into labels or preferences rather than synthesize new content.

A dense vector that represents a token or other discrete item so the model can run continuous math on it.

Qualitative capability jumps that appear only past certain model scale or training thresholds—distinct from smooth metric improvements tracked by scaling laws.

A network stack that maps raw or patched inputs into internal representations—tokens, latents, or context vectors—for downstream decoders or heads.

An architecture that compresses input into representations with an encoder, then generates outputs with a decoder—often via cross-attention between the two stacks.

A measure of how spread out or uncertain a probability distribution is over next-token choices.

A large pretrained model intended as a shared starting point for many downstream tasks via fine-tuning or prompting.

How well a model performs on data it was not explicitly trained to memorize—validation sets, new domains, and live traffic are the practical tests.

A model trained to produce new outputs such as text, images, or audio rather than only assign labels to fixed categories.

The direction and size of change each parameter should take to reduce the training loss.

A deterministic next-token rule that always picks the highest-probability option, making output stable and repeatable but usually less diverse.

The width of a model's internal vectors—the number of dimensions in each token embedding and each token's per-position hidden state before the vocabulary projection.

The serving latency metric that measures the delay between streamed output tokens after generation has started.

The saved key and value tensors a decoder keeps from earlier tokens so later tokens can reuse attention state instead of recomputing the full prefix.

A compressed internal code that summarizes input structure for reconstruction, generation, or downstream tasks.

A raw, unnormalized score for each vocabulary item before softmax turns scores into probabilities.

A scalar score that measures how wrong the model's predictions are so training knows which direction to improve.

The kind of input or output data a model is built to handle, such as text, images, audio, or video.

A trained machine learning system that maps inputs to outputs using learned parameters.

How much function a model can represent—driven by parameter count, width, depth, and training data—before optimization and regularization constrain what it actually learns.

A reusable building block inside a model architecture, such as attention or a feed-forward network.

Layers that rescale hidden vectors inside each transformer block so activations stay stable as depth and batch composition change.

Extra memory an adaptive optimizer keeps beyond model weights, such as momentum and variance estimates for each parameter.

When a model fits training noise so closely that held-out or production data perform worse—the classic gap between memorization and useful generalization.

A learnable numeric value stored in the model weights that training updates to fit data.

A fixed-size chunk of input data—pixels, spectrogram frames, or grouped tokens—that an encoder turns into a representation vector.

A standard language-model evaluation metric: exponentiated average cross-entropy over tokens, lower is better when comparing models on the same corpus.

An internal encoding of input data that downstream layers or tasks use instead of raw features.

Skip paths that add each sublayer's output back onto the running hidden stream so deep transformer stacks stay trainable.

The next-token decision step that turns a probability distribution into one chosen token, shaping how diverse, stable, or controllable a model's output feels.

Empirical power-law relationships between model scale, compute, data, and downstream loss or capability—used to forecast training budgets and benchmark trends.

A broader shortcut path that carries information around one or more layers, including the residual adds used inside transformers.

A function that turns a vector of logits into a probability distribution that sums to one.

Reserved tokenizer markers such as beginning-of-sequence, end-of-sequence, padding, and separators that carry structural meaning instead of ordinary text meaning.

A positive scaling factor applied to logits before softmax to make the next-token distribution sharper or flatter.

A multi-dimensional array of numbers that carries activations, weights, or gradients through the network.

The serving tradeoff between aggregate completed work over time and the wait experienced by an individual request under concurrency.

The serving latency metric that measures how long a reader waits from request start until the first generated token appears.

The smallest unit of text a language model reads and predicts—usually a word piece, not always a whole word. Each token ID maps to a dense vector through vector embedding of model hidden size before attention runs.

Tokens per second (tokens/s, tok/s, or TPS) is a throughput rate: how many model tokens a serving system generates per second over the measurement window you choose.

A next-token rule that keeps only the k highest-probability options before drawing one, trading some stability for more bounded diversity than greedy decoding.

A next-token rule that keeps the smallest probability mass above a threshold before drawing one token, adapting the candidate count from step to step.

An ordered list of numbers that represents a point or direction in continuous space—embeddings and activations are vectors at different stages of the model.

The total number of token IDs a tokenizer can emit, including ordinary text pieces and reserved control tokens that occupy part of the same table.