Sinusoidal positional embeddings compute fixed wave-shaped position vectors from token index, so models get absolute order without learning a separate parameter row for each position.
Sinusoidal positional embeddings are an absolute position method that computes each position vector from repeating sine and cosine patterns at different frequencies. Position 1, position 2, position 3, and later positions all get deterministic vectors from the same formula, and those vectors are added to tokenembeddings before attention begins.
Why It Matters
The original Transformer used sinusoidal positions because they gave the model a built-in sense of order without learning a separate parameter row for every index. Since the pattern is generated rather than stored in a learned table, it stays well-defined for positions beyond the exact indices seen during training, even though that does not guarantee strong long-context quality by itself.
Simple Example
Imagine plotting one slow wave and one fast wave for each token position. Position 5 and position 6 get slightly different values on both waves, so their combined vectors are distinct. When those vectors are added to the tokenembeddings, attention can tell that the same word appearing at position 5 is not in exactly the same place as the same word at position 6.