T5
An encoder-decoder transformer from Google Research that reformulates every NLP task as reading text and writing text in a shared token space.
Text-to-Text Transfer Transformer (T5) is a transformer model family that treats translation, summarization, classification, and similar jobs as the same interface: read a text prompt, run it through an encoder that builds context, then let a decoder generate the answer as more text.
At a glance
- Family
- t5
- Source type
- Open weights
- Released
- October 2019
- Authors
- Colin Raffel, Noam Shazeer, Adam Roberts, et al.
- Modalities
- Text
- Parameter count
- 60M to 11B parameters (family variants)
- Context length
- 512 tokens
What It Is
Text-to-Text Transfer Transformer (T5) is an encoder-decoder transformer model family from Google Research. Instead of building a separate model head for every task, T5 reformulates natural language problems as text-to-text: the model always reads text and writes text. That unified interface made T5 a central reference for transfer learning because the same pretrained weights can be steered toward many downstream jobs by changing only the input and target wording.Inputs And Outputs
T5 reads tokenized text and produces tokenized text. Input and output share one SentencePiece vocabulary, so a classification label, a translation, or a summary is just another text string on the output side. In practice, each task uses a short text prefix or template in the input—such as "translate English to German:" or "summarize:"—so the model knows which behavior to apply while the token interface stays the same.Architecture
T5 follows an encoder-decoder layout. The encoder stack reads the full input sequence with bidirectional self-attention so every token can gather context from the whole prompt. The decoder stack then generates output tokens one at a time: masked self-attention keeps each new token from peeking ahead, and cross-attention lets decoder states read the encoder's hidden states. T5 also replaces absolute position embeddings with relative position bias, so attention scores depend on token distance rather than a fixed position vector added to every embedding.Output
TextLinearRepeated decoder block containerDecoder attention sublayer containerDecoder feed-forward sublayer containerN×Add & NormMasked
Multi-Head
AttentionAdd & NormFeed
ForwardCross
AttentionRepeated encoder block containerEncoder attention sublayer containerEncoder feed-forward sublayer containerN×Add & NormBidirectional
Multi-Head
AttentionRelative
Position
BiasAdd & NormFeed
ForwardInput
EmbeddingSentencePiece
TokenizationInput
TextLinear to Output
TextRepeated decoder block container to LinearCross
Attention to Repeated decoder block containerRepeated encoder block container to Cross
AttentionInput
Embedding to Repeated encoder block containerSentencePiece
Tokenization to Input
EmbeddingInput
Text to SentencePiece
TokenizationMasked
Multi-Head
Attention to Add & NormAdd & Norm to Feed
ForwardFeed
Forward to Add & NormAdd & Norm to Add & NormCross
Attention to Masked
Multi-Head
AttentionBidirectional
Multi-Head
Attention to Add & NormAdd & Norm to Feed
ForwardFeed
Forward to Add & NormAdd & Norm to Add & NormRelative
Position
Bias to Bidirectional
Multi-Head
AttentionInput
Embedding to Bidirectional
Multi-Head
Attention
Important Modules
The registry-linked modules below are the main pieces that define T5's behavior. SentencePiece tokenization turns raw text into shared input and output pieces. Relative position bias supplies distance-aware attention inside both stacks. Multi-head attention moves information across tokens, cross-attention connects encoder context to the decoder, layer normalization stabilizes each block, and the standard feed-forward network applies a local dense transform at every position.Training
T5 pretraining uses a denoising objective called span corruption: random contiguous spans in the input text are replaced with sentinel tokens, and the model learns to reconstruct the missing pieces as its text output. That stage runs on large cleaned web text, after which supervised fine-tuning adapts the same weights to downstream benchmarks by formatting each example as a text-to-text pair. The linked training regimes and reference below point to the primary records for those stages.Training regimes
Linked papers
No linked paper pages listed yet.