How a token reads other tokens from the same sequence so a transformer can mix context before producing the next representation.
Self-attention lets each token compare itself with other tokens from the same sequence, decide which positions matter, and blend that context into an updated representation. It is the broad transformer idea behind many later attention variants, not just one specialized optimization.
What It Is
Self-attention is the case where queries, keys, and values all come from the same running sequence. A token asks what it needs, compares that question against other token positions, and then mixes in information from the positions that score highest. In a decoder model, the mask usually limits each token to earlier positions. In an encoder, the same basic mechanism can look both backward and forward.
Why It Matters
This is the mechanism that makes transformers good at context mixing. Instead of reading text through one fixed window, the model can decide that one earlier word matters more than another and weight them differently for each token. Once you understand broad self-attention, pages like multi-head attention and grouped-query attention become easier to read because they are changing how this baseline is expressed, stored, or scaled.
Simple Example
In the sentence "The animal didn't fit in the suitcase because it was too small," the token for "it" needs clues from earlier tokens in the same sentence. Self-attention lets that token score earlier positions such as "animal" and "suitcase," then blend the most useful evidence into its next hidden state before the transformer block moves on.
Inside A Transformer Block
In a transformer architecture, each token first arrives as a tokenvector. The self-attention step compares that token's query against keys from the same sequence, mixes the most relevant values, and hands the updated representation to the rest of the block. Decoder-only models usually apply a causal mask so the token can only read earlier positions, while encoder-style stacks often let the token look in both directions.
Full self-attention gives every token a chance to consult every other token in the same sequence, but that flexibility is expensive. As sequences grow, the score matrix and the saved key-value state grow quickly, which is why long-context systems often reach for memory-saving variants, sparse patterns, or serving tricks. Self-attention also does not guarantee good reasoning by itself: the model can still spread focus across too many irrelevant positions or miss the one clue that matters.