Positional encodings
How transformers inject order into otherwise permutation-invariant attention through absolute, relative, and rotary position schemes.
Attention treats tokens as an unordered set unless you add position information, so this hub shows how absolute embeddings, relative biases, rotary methods, and no-position experiments each restore sequence order in a different way.
What It Is
Positional encodings tell a transformer where each token sits in the sequence. Self-attention scores token pairs without built-in order, so swapping positions would otherwise look the same to the model. Early transformers solved that by adding absolute position vectors to token embeddings. Later families moved position into attention itself through relative biases, rotary position embedding (RoPE), or distance-based bias methods such as ALiBi.Why It Matters
The position scheme shapes how a model remembers order, how gracefully it stretches past training length, and which long-context tricks later become possible. Absolute embeddings are easy to reason about but tie positions to fixed tables. Relative and rotary methods keep distance information inside attention, which many modern decoders prefer. A few papers also test NoPE, which removes explicit positional encoding to see what order signal attention can recover on its own.Family Split
The main branches are easy to sort. Absolute positional embeddings attach an index-specific vector to each token. Relative position bias changes attention scores based on token distance. RoPE rotates query and key features so relative offsets appear inside the dot product. ALiBi adds a simpler distance penalty directly to logits. NoPE is the unusual baseline where a model tries to work without an explicit position mechanism at all.Common Confusions
Positional encoding is not the same as token embedding: embeddings carry vocabulary identity while position schemes carry index or distance. RoPE and ALiBi are not separate attention variants; they only change how position enters the attention sublayer. NoPE also does not mean long-context support by itself. It usually names an ablation or architectural choice where explicit position handling is removed rather than improved.References
- Vaswani, Ashish, et al. "Attention Is All You Need." arXiv, 2017, https://arxiv.org/abs/1706.03762.
- Su, Jianlin, et al. "RoFormer: Enhanced Transformer with Rotary Position Embedding." arXiv, 2021, https://arxiv.org/abs/2104.09864.
- Press, Ofir, et al. "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation." arXiv, 2021, https://arxiv.org/abs/2108.12409.