BERT model

An encoder-only transformer that learns bidirectional text representations through masked language modeling pretraining.

Bidirectional Encoder Representations from Transformers (BERT) is an encoder-only transformer that reads whole token sequences at once and learns contextual vectors for each position, which is why it became the standard starting point for classification and understanding tasks rather than open-ended text generation.

At a glance

Family
bert
Source type
Research
Released
October 2018
Authors
Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al.
Source
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Modalities
Text
Parameter count
110 million to 340 million parameters
Context length
512 tokens
Precision
float32

What It Is

Bidirectional Encoder Representations from Transformers (BERT) is a research text model from Google that popularized deep bidirectional transformer encoders. It is encoder-only: it reads an input sequence and produces contextual representations for every token instead of generating new text one token at a time like decoder-only language models.

Inputs And Outputs

BERT takes tokenized text as input. WordPiece tokenization splits words into subword pieces so rare words can still be represented. Each token receives a learned embedding, a position embedding that marks its place in the sequence, and a segment embedding when two sentences are packed together for tasks like question answering. The model returns a hidden vector for each token position. Downstream systems use those vectors for sentence classification, question answering, named-entity recognition, and other understanding tasks rather than open-ended generation.

Architecture

Unlike decoder-only models that predict the next token from left to right, BERT processes the full sequence through transformer encoder blocks with bidirectional self-attention. The diagram below traces WordPiece tokens into combined embeddings, a repeated encoder block with bidirectional attention and feed-forward layers, contextual token vectors, and the masked-language-model or task heads used during pretraining and fine-tuning.

Important Modules

The modules below explain most of BERT's mechanics. WordPiece handles subword tokenization, bidirectional attention lets each token see both sides of the sequence, and learned positional embeddings mark token order inside the encoder stack.

Training

BERT's defining pretraining recipe combines masked language modeling with next-sentence prediction over large text corpora. In masked language modeling, random input tokens are replaced with a mask symbol and the model learns to predict the original token using bidirectional context from both sides of the gap. That objective teaches the encoder rich representations before task-specific fine-tuning on labeled data.

Training regimes

Linked papers

Practical Notes

BERT is widely reused as a feature extractor or fine-tuning backbone for understanding tasks. It is usually not the right base model for open-ended generation because it lacks an autoregressive decoder, and its 512-token context window is short compared with later long-context language models.

Tags

References

  1. Devlin, Jacob, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv, 2018, https://arxiv.org/abs/1810.04805.

U-Net

Previous Page

CLIP

Next Page

On this page

What It IsInputs And OutputsArchitectureImportant ModulesTrainingPractical NotesRelated Models, Modules, And PapersTagsReferences