A learned vector that turns discrete tokens, image patches, or other inputs into numbers a model can process.
Embeddings are learned vectors that give discrete inputs a numeric form, and most transformer stacks split that job into tokenembeddings for what each input is and positional embeddings for where it sits—together they form an embedding space you can reason about without treating every coordinate as literal human meaning.
What It Is
An embedding is a fixed-size list of numbers (a vector) tied to a discrete input such as a text token, an image patch, or another symbol ID. During training the model learns those vectors so later layers can run continuous math on them. In language models, the token embedding layer looks up one vector per token ID: that vector encodes vocabulary identity or content—what the symbol is—rather than where it appears in the sequence.
Why It Matters
Embeddings are where raw inputs become geometry the network can mix. After training, inputs with similar roles often end up with nearby vectors, so it helps to picture an embedding space: a high-dimensional map where distance is a useful hint about learned relationships. That space is a modeling convenience, not a claim that each axis names a human concept. Every later layer—attention, feed-forward blocks, and output heads—operates on these vectors, so deployment sizing also counts how many embedding rows and dimensions you store.
Simple Example
Suppose a tokenizer turns a three-word sentence into token IDs 12, 42, and 7. The token embedding table maps each ID to a 768-number vector, giving a 3×768 tensor of content vectors. Before attention runs, many GPT-style models add learned positional embeddings at each index—one pattern for position 0, another for position 1, and so on—so the stack knows order as well as identity. The sum (or equivalent combination) at each position is what the first transformer block receives.
Where It Appears
Tokenembeddings sit at the bottom of decoder-only transformers such as GPT-3, encoder inputs for sequence-to-sequence models, and retrieval systems that compare query and document vectors. Positional information is usually fused with tokenembeddings before attention, whether through learned absolute tables, sinusoidal patterns, or schemes that inject order inside attention instead. Use the related docs below to continue from tokenization into positional encodings and representative model stacks.
Common Confusions
Tokenembeddings and positional embeddings solve different problems: tokenvectors carry what the input is, while positional vectors (or position-aware attention biases) carry where it sits. Embedding space is a helpful way to talk about similarity between learned vectors, but it is not the same as everyday word meaning—coordinates are whatever training made useful for the task. An embedding is also not a one-hot vector: the table stores dense learned weights rather than a single 1 in a huge sparse vector. Embeddings are inputs to the stack, not logits or probabilities from the vocabulary head.