Position Encoding

Meta's open-weight Llama-family decoder-only transformer release, spanning dense and instruction-tuned checkpoints with long-context support.

An encoder-decoder transformer from Google Research that reformulates every NLP task as reading text and writing text in a shared token space.

Absolute positional embeddings give each token position its own index-specific vector, then add that vector directly to the token embedding before attention runs.

Attention with linear biases adds distance-based penalties to attention logits so models can extrapolate to longer sequences without explicit position embeddings.

Learned positional embeddings train a position table alongside token embeddings, so each index gets a position vector the model can adapt during training.

LongRoPE is a RoPE extension method that rescales rotary dimensions and positions non-uniformly so a model can stretch much farther into long-context ranges.

NoPE means a model removes explicit positional encoding and tests how much order signal attention and training can recover without a dedicated position mechanism.

NTK-aware RoPE scaling changes the rotary frequency base so a RoPE model keeps more useful relative behavior when it is asked to run beyond its trained context length.

Positional interpolation extends a RoPE model by compressing new token positions back into the position range the model was originally trained to handle.

Relative position bias changes attention scores with distance-aware bias terms so the model reasons about how far apart tokens are instead of storing a fixed vector for every position.

Rotary position embedding rotates query and key vectors by token index so attention scores depend on relative distance, as used in Llama-class decoder models.

Sinusoidal positional embeddings compute fixed wave-shaped position vectors from token index, so models get absolute order without learning a separate parameter row for each position.

SuperHOT RoPE stretches rotary positions across a longer range, then fine-tunes the model so the compressed position map still behaves well at the new window.

T5 relative position bias buckets token distances into ranges, then adds a learned bias per bucket so attention can reason about near and far positions without a full absolute table.

YaRN is a RoPE extension method that mixes interpolation-style scaling with continued training so a model can use a larger context window more efficiently.

A positional idea that subtracts larger attention penalties for farther token pairs instead of storing explicit position embeddings.

How transformers inject order into otherwise permutation-invariant attention through absolute, relative, and rotary position schemes.