Browse the Atlas

A large decoder-only language model that made in-context prompting and few-shot text generation a mainstream reference point.

An encoder-only transformer that learns bidirectional text representations through masked language modeling pretraining.

A dual-encoder multimodal model that learns to align text and images in one shared embedding space.

Open omnimodal world model family from NVIDIA for physical AI, combining autoregressive reasoning with diffusion-based generation across text, image, video, audio, and action outputs through Nano and Super checkpoints.

A lighter DeepSeek-V4 serving profile that keeps the same conceptual stack while prioritizing faster deployment paths.

The larger DeepSeek-V4 reasoning model, built around long-context attention, MoE routing, and coupled post-training and serving choices.

A model family that learns environment dynamics or state transitions so it can predict or simulate what happens next.

A model trained to produce new outputs such as text, images, or audio rather than only assign labels to fixed categories.

A model family that accepts or produces more than one data modality, such as text paired with images or audio.

A generation paradigm that produces outputs one discrete step at a time, each step conditioned on everything generated so far.

A large pretrained model intended as a shared starting point for many downstream tasks via fine-tuning or prompting.

A model trained to score, rank, or classify inputs into labels or preferences rather than synthesize new content.

The kind of input or output data a model is built to handle, such as text, images, audio, or video.

A generative model family that learns to reverse a noise corruption process through many iterative denoising steps.

A network stack that maps raw or patched inputs into internal representations—tokens, latents, or context vectors—for downstream decoders or heads.

A network stack that turns internal representations into outputs—tokens, pixels, or structured predictions—often one step at a time with attention to prior context.

An architecture that compresses input into representations with an encoder, then generates outputs with a decoder—often via cross-attention between the two stacks.

An attention variant that reduces key-value cache memory by sharing key-value heads across query groups.

How transformer blocks mix information across token positions using query, key, and value projections.

A gated feed-forward design that multiplies a value branch by a swish gate before projecting back.

A simple activation function that keeps positive values and turns negative values into zero.

The baseline attention design that gives every query head its own key-value head pair.

A neural network whose signals move from input to output through weighted layers without recurrent loops back into earlier layers.

A function that turns a vector of logits into a probability distribution that sums to one.

Skip paths that add each sublayer's output back onto the running hidden stream so deep transformer stacks stay trainable.

A broader shortcut path that carries information around one or more layers, including the residual adds used inside transformers.

The numeric output of a layer after its linear transform and nonlinearity, passed forward through the network.

Layers that rescale hidden vectors inside each transformer block so activations stay stable as depth and batch composition change.

A dense vector that represents a token or other discrete item so the model can run continuous math on it.

A raw, unnormalized score for each vocabulary item before softmax turns scores into probabilities.

A multi-dimensional array of numbers that carries activations, weights, or gradients through the network.

An ordered list of numbers that represents a point or direction in continuous space—embeddings and activations are vectors at different stages of the model.

The total number of token IDs a tokenizer can emit, including ordinary text pieces and reserved control tokens that occupy part of the same table.

How attention, feed-forward layers, normalization, residuals, and position information repeat inside each decoder-style transformer block.

How transformers inject order into otherwise permutation-invariant attention through absolute, relative, and rotary position schemes.

Common techniques that stretch usable context beyond a model's original training window without rebuilding the entire architecture from scratch.

Storing or computing with fewer bits so models use less memory and sometimes run more efficiently, with tradeoffs around quality, calibration, and hardware support.

Why stretching sequence length strains attention compute, KV-cache memory, position schemes, and retrieval quality even when a model advertises a large context window.

Compressing the cached keys and values kept across autoregressive decode steps to reduce long-context serving memory.

The next-token decision step that turns a probability distribution into one chosen token, shaping how diverse, stable, or controllable a model's output feels.

A next-token rule that keeps only the k highest-probability options before drawing one, trading some stability for more bounded diversity than greedy decoding.

A next-token rule that keeps the smallest probability mass above a threshold before drawing one token, adapting the candidate count from step to step.

A deterministic next-token rule that always picks the highest-probability option, making output stable and repeatable but usually less diverse.

A positive scaling factor applied to logits before softmax to make the next-token distribution sharper or flatter.

The repeated next-token stage that reuses the KV cache and turns inter-token latency into the main reader experience after prefill finishes.

The saved key and value tensors a decoder keeps from earlier tokens so later tokens can reuse attention state instead of recomputing the full prefix.

The serving latency metric that measures how long a reader waits from request start until the first generated token appears.

The serving latency metric that measures the delay between streamed output tokens after generation has started.

A technical report that bundles DeepSeek-V4 architecture, long-context attention, post-training methods, and serving systems into one connected release.

The 2017 paper that introduced the transformer, using self-attention as the main way tokens mix information in encoder-decoder sequence models.

The paper that introduced deep bidirectional transformer encoders trained with masked language modeling and fine-tuned for language understanding tasks.

A technical report that introduced GPT-2 as a large decoder-only language model trained with broad next-token pretraining and byte-level BPE tokenization.

The research paper behind Stable Diffusion-style image generation: compress images into a latent space, denoise there instead of in pixel space, then decode the result.

The CLIP paper introduced contrastive language-image pretraining with separate image and text encoders that map paired captions and images into one shared embedding space.

A DeepSeek-V4 post-training routine where the student is refined on trajectories produced under its current policy rather than only on frozen offline traces.

A DeepSeek-V4 post-training pattern that sharpens separate capability areas before they are merged back into the final stack.

A DeepSeek-V4 training step that prepares the model for very low precision by exposing it to quantization effects during training.

The pretraining-style regime where a denoiser learns to predict noise or a clean-direction signal from corrupted examples across a noise schedule.

A preference-optimization training regime that teaches a model from preferred versus rejected answer pairs without fitting a separate reward model and PPO loop first.

A teacher-student training regime where a smaller model learns from supervision produced by a larger or stronger teacher model.

A serving system practice that turns a trained model plus its runtime stack into a live, versioned inference service on real hardware.

A serving system that decides which model, hardware tier, or execution path should handle a request so operators can balance cost, latency, safety, and specialization.

A serving system that spills part of the KV cache to disk so very long sessions remain workable without keeping every token state in expensive fast memory.

A serving system that overlaps expert routing, communication, and compute so sparse MoE execution does not stall on movement alone.

A serving system that groups multiple requests or decode steps together so accelerators stay busier, raising throughput while often adding wait time for each individual request.

A serving system that keeps an active decode set refilled as requests finish or become compatible, so accelerators spend less time waiting for a whole batch to turn over.

The smallest unit of text a language model reads and predicts—usually a word piece, not always a whole word. Each token ID maps to a dense vector through vector embedding of model hidden size before attention runs.

The blueprint that defines how a model's layers, modules, and data flow connect during inference.

Post-training steps that steer a model toward helpful, honest, and policy-compliant behavior—often via preference data, reward models, or safety filters.

The backward pass that applies the chain rule on a computational graph to compute gradients for every differentiable parameter.

A finer-grained part inside a module, such as an attention head or projection layer.

A record of which operations produced each tensor so automatic differentiation can run backward.