Architecture

The post-pretraining goal of shaping a broadly capable base model toward helpful, safer, preference-following behavior before or during deployment.

Post-training steps that steer a model toward helpful, honest, and policy-compliant behavior—often via preference data, reward models, or safety filters.

The blueprint that defines how a model's layers, modules, and data flow connect during inference.

A positional idea that subtracts larger attention penalties for farther token pairs instead of storing explicit position embeddings.

A generation paradigm that produces outputs one discrete step at a time, each step conditioned on everything generated so far.

A diffusion inference technique that steers denoising by comparing prompt-conditioned and prompt-free predictions, then blending them with a guidance scale.

A finer-grained part inside a module, such as an attention head or projection layer.

A record of which operations produced each tensor so automatic differentiation can run backward.

Extra inputs—text prompts, class labels, images, or guidance weights—that steer a generative model without retraining the full parameter stack each time.

Common techniques that stretch usable context beyond a model's original training window without rebuilding the entire architecture from scratch.

The maximum number of tokens a model can attend over in one forward pass, distinct from training sequence length and how many tokens it may emit in a session.

The repeated next-token stage that reuses the KV cache and turns inter-token latency into the main reader experience after prefill finishes.

A network stack that turns internal representations into outputs—tokens, pixels, or structured predictions—often one step at a time with attention to prior context.

A generation paradigm that starts from noisy latents and iteratively removes noise—or adds structure—until a clean sample emerges.

A generative model family that learns to reverse a noise corruption process through many iterative denoising steps.

A model trained to score, rank, or classify inputs into labels or preferences rather than synthesize new content.

A learned vector that turns discrete tokens, image patches, or other inputs into numbers a model can process.

A dense vector that represents a token or other discrete item so the model can run continuous math on it.

Qualitative capability jumps that appear only past certain model scale or training thresholds—distinct from smooth metric improvements tracked by scaling laws.

A network stack that maps raw or patched inputs into internal representations—tokens, latents, or context vectors—for downstream decoders or heads.

An architecture that compresses input into representations with an encoder, then generates outputs with a decoder—often via cross-attention between the two stacks.

Floating-point operations counted to estimate how much arithmetic work a model needs during inference, before teams compare serving cost, throughput plans, or hardware capability.

A generative training objective that teaches a model how samples should move from a simple or noisy starting state toward real data.

A large pretrained model intended as a shared starting point for many downstream tasks via fine-tuning or prompting.

How well a model performs on data it was not explicitly trained to memorize—validation sets, new domains, and live traffic are the practical tests.

A model trained to produce new outputs such as text, images, or audio rather than only assign labels to fixed categories.

The serving latency metric that measures the delay between streamed output tokens after generation has started.

Saved key and value tensors from earlier tokens that let later generation steps reuse attention state instead of recomputing the full prefix.

The saved key and value tensors a decoder keeps from earlier tokens so later tokens can reuse attention state instead of recomputing the full prefix.

A compressed internal code that summarizes input structure for reconstruction, generation, or downstream tasks.

A learned compressed representation space where models store internal codes instead of raw pixels or token ids, then decode or denoise back to visible outputs.

How quickly weights, activations, and KV cache bytes can move between memory and compute during model serving, and why that movement can cap useful throughput.

A broad sparse-routing architecture pattern that activates only a few expert sub-networks per token instead of one dense path for every token.

The kind of input or output data a model is built to handle, such as text, images, audio, or video.

A trained machine learning system that maps inputs to outputs using learned parameters.

How much function a model can represent—driven by parameter count, width, depth, and training data—before optimization and regularization constrain what it actually learns.

A reusable building block inside a model architecture, such as attention or a feed-forward network.

A model family that accepts or produces more than one data modality, such as text paired with images or audio.

Keeping activations on a predictable scale so deep model stacks stay trainable and their sublayers receive stable inputs.

Layers that rescale hidden vectors inside each transformer block so activations stay stable as depth and batch composition change.

Learning from examples or trajectories produced by the model or agent policy that is currently being updated.

When a model fits training noise so closely that held-out or production data perform worse—the classic gap between memorization and useful generalization.

A small generated concept page that demonstrates the page-spec bundle workflow end to end.

A fixed-size chunk of input data—pixels, spectrogram frames, or grouped tokens—that an encoder turns into a representation vector.

A standard language-model evaluation metric: exponentiated average cross-entropy over tokens, lower is better when comparing models on the same corpus.

How transformers inject order into otherwise permutation-invariant attention through absolute, relative, and rotary position schemes.

The prompt-processing stage that reads the full input once, builds the first attention state, and sets up generation before any reply token appears.

The serving setup that separates prompt processing from token-by-token generation because the two stages stress hardware, memory, and latency in different ways.

Training-time and model-design choices that discourage brittle memorization so a model is more likely to perform well on new examples.

An internal encoding of input data that downstream layers or tasks use instead of raw features.

Skip paths that add each sublayer's output back onto the running hidden stream so deep transformer stacks stay trainable.

A throughput-ceiling chart that shows when memory movement or arithmetic compute limits attainable performance, using arithmetic intensity to explain which bound is active.

Empirical power-law relationships between model scale, compute, data, and downstream loss or capability—used to forecast training budgets and benchmark trends.

How a token reads other tokens from the same sequence so a transformer can mix context before producing the next representation.

A broader shortcut path that carries information around one or more layers, including the residual adds used inside transformers.

Reserved tokenizer markers such as beginning-of-sequence, end-of-sequence, padding, and separators that carry structural meaning instead of ordinary text meaning.

Training material produced or transformed by one model to teach another, distinct from raw web corpora collected for pretraining or demonstrations written directly by people.

Extra prompt-derived vectors that steer an image generator while the denoiser removes noise step by step.

The serving tradeoff between aggregate completed work over time and the wait experienced by an individual request under concurrency.

The serving latency metric that measures how long a reader waits from request start until the first generated token appears.

The smallest unit of text a language model reads and predicts—usually a word piece, not always a whole word. Each token ID maps to a dense vector through vector embedding of model hidden size before attention runs.

How tokenizers turn raw text into reusable pieces before embeddings, attention, and next-token prediction begin.

Tokens per second (tokens/s, tok/s, or TPS) is a throughput rate: how many model tokens a serving system generates per second over the measurement window you choose.

A model family built from stacked attention and feed-forward blocks that mix information across token positions in parallel.

How attention, feed-forward layers, normalization, residuals, and position information repeat inside each decoder-style transformer block.

Models that produce moving visual sequences by generating or refining frames—or visual tokens arranged over time—while keeping motion, identity, and appearance coherent from one moment to the next.

How images and video frames become patches, discrete codes, or latent tokens so sequence-based models can read visual data.

Why stretching sequence length strains attention compute, KV-cache memory, position schemes, and retrieval quality even when a model advertises a large context window.

A model family that learns environment dynamics or state transitions so it can predict or simulate what happens next.