BERT Paper

The paper that introduced deep bidirectional transformer encoders trained with masked language modeling and fine-tuned for language understanding tasks.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding became a reference point because it showed how a deep bidirectional encoder could learn general language representations from unlabeled text and then adapt to many downstream tasks through fine-tuning.

At a glance

Published

October 2018

Authors

Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al.

Introduces

No introduced records listed yet.

Why It Matters

Before BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, many language models read text in one direction and learned by predicting the next token. That pattern fits generation well, but it limits how much surrounding context each position can use during pretraining. BERT helped establish the encoder-only path: pretrain a bidirectional encoder on broad text with masked language modeling, then fine-tune the same stack for classification, question answering, and other understanding tasks. That split between broad pretraining and task-specific adaptation became a common template for later encoder models without cataloguing every BERT checkpoint variant.

Method Or Architecture

The paper keeps the transformer architecture in an encoder-only form rather than a decoder that generates left to right. WordPiece tokenization turns raw text into subword units, embeddings carry token and position information into the stack, and stacked encoder blocks apply bidirectional attention so each token can use context from both sides. Each encoder block also uses a feed-forward layer with gelu activation between attention passes. Pretraining uses masked language modeling: randomly hide some input tokens and train the model to predict those hidden tokens from both left and right context. Unlike next-token pretraining, the model is not limited to looking only backward through the sequence. A companion next-sentence objective asks whether one sentence truly follows another, which helps the encoder reason over sentence pairs. Together these choices produce a reusable bidirectional encoder representation that downstream tasks can fine-tune instead of training a fresh model from scratch.
The BERT paper combines WordPiece inputs, transformer encoder blocks with bidirectional attention, masked language modeling pretraining, and encoder-only fine-tuning for downstream understanding tasks.

Evidence

The paper's central evidence is that one pretrained bidirectional encoder can transfer across sentence-pair classification, single-sentence tagging, question answering, and related understanding tasks after relatively small task-specific fine-tuning. That pattern helped make encoder pretraining plus fine-tuning a standard recipe for language understanding work rather than training a separate model from scratch for every task.

Limitations

The paper focuses on understanding-style fine-tuning rather than open-ended text generation, and several design choices such as the next-sentence objective were later refined or replaced in successor models. It also predates many later advances in scaling laws, instruction tuning, and long-context training, so readers should treat it as a foundational encoder-only reference point rather than the final word on modern language-model design.

Tags

References

  1. Devlin, Jacob, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv, 2018, https://arxiv.org/abs/1810.04805.

Attention Is All You Need

Previous Page

DeepSeek-V4

Next Page

On this page

Why It MattersMethod Or ArchitectureEvidenceLimitationsRelated ToTagsReferences