How transformer blocks mix information across token positions using query, key, and value projections.
Attention lets each token decide which other tokens matter most, scoring possible connections and blending information from the most relevant positions so context can move dynamically across a sequence.
Attention is the mechanism that lets one token read information from other tokens. Each position produces a query, key, and value. The query asks what to look for, the keys describe what each position offers, and the values carry the information that gets mixed into the output.
Why It Exists
Attention optimizes for flexible context mixing. Instead of using a fixed local rule, it lets the model decide which positions matter for the current token and how strongly they should contribute.
How It Works
For each token, the model compares that token's query against the keys from other positions to produce attention scores. Those scores are normalized into weights, and the model uses the weights to blend the value vectors into one updated token state.
Compared To Nearby Modules
Feed-forward layers refine one token state at a time after context has already been gathered. Attention is the part that moves information across positions in the first place. Many attention variants then change either head sharing, sequence reach, or compute scaling.
Example Architectures
Attention appears in decoder-only language models, encoder models, encoder-decoder systems, and many multimodal transformers because all of them need a way to move information across positions.
Limitations And Tradeoffs
Dense attention can become expensive as sequence length grows because every token may compare against many others. That is why later variants often trade memory, bandwidth, or global reach for better scaling.
Why It Still Matters
Attention remains the reference point for understanding modern sequence models. When a paper introduces a new attention variant, the real question is usually which part of the baseline mechanism it changed.