Attention Is All You Need
The 2017 paper that introduced the transformer, using self-attention as the main way tokens mix information in encoder-decoder sequence models.
Attention Is All You Need introduced the transformer as a sequence model built around self-attention instead of recurrence or convolutions. Tokens exchange information through attention layers stacked in an encoder and a decoder, which became the reusable backbone behind most later large language models.
At a glance
Published
June 2017
Authors
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.
Supports
No supported records listed yet.
Why It Matters
Before this paper, many sequence models moved information step by step. Recurrent neural networks passed a hidden state forward one token at a time, so distant tokens could only influence each other through many sequential hops. Convolutional sequence models mixed information through local windows, which meant long-range links often required deep stacks of layers. Attention Is All You Need replaced those paths with self-attention, where each token can look directly at other tokens and update its representation in parallel. That shift removed the sequential bottleneck that slowed training on long sequences and made it easier to reuse the same stack for translation, language modeling, and later decoder-only systems. The paper turned the transformer into the reference architecture readers still meet first when learning modern language models.Method Or Architecture
The transformer organizes sequence processing into an encoder stack and a decoder stack. The encoder reads the full input sequence and builds representations each output step can draw from. The decoder generates the output sequence while attending both to its own prior tokens and to the encoder output. Self-attention is the main mixing mechanism inside each layer: every token forms weighted connections to other tokens instead of waiting for information to travel through recurrence or a fixed convolution window. Multi-head attention runs several of those connection patterns in parallel so the model can track different kinds of relationships at once, such as syntax, agreement, or long-distance references. Positional encodings add order information because attention by itself treats tokens as an unordered set. After attention mixes information across positions, feed-forward blocks apply the same small nonlinear transformation independently to each token position. Residual or skip connections add each sublayer's input back to its output so gradients can flow through deep stacks during training. Together, those pieces define a repeatable layer pattern—attention, feed-forward processing, and residual paths—that later model families could copy, trim, or extend without returning to recurrence as the core mixer.Attention Is All You NeedAttention-centered architectureMulti-head attentionPositional encodingsEncoder-decoder stackFeed-forward sublayersResidual/skip pathsAttention Is All You Need to Attention-centered architectureAttention Is All You Need to Multi-head attentionAttention Is All You Need to Positional encodingsAttention Is All You Need to Encoder-decoder stackAttention Is All You Need to Feed-forward sublayersAttention Is All You Need to Residual/skip paths