AIAny
Natural Language Processing Papers2018

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Pre-trains a deep bidirectional Transformer encoder with masked-language-modeling and next-sentence prediction, then fine-tunes one model on 11 NLP tasks, reaching state-of-the-art on GLUE, SQuAD, and MultiNLI with little task-specific tuning.

Visit Website

Introduction

Before this paper, transfer learning in NLP mostly meant feeding pre-trained word vectors or a left-to-right language model into a task-specific architecture you still had to design. The reframing here is that almost none of that scaffolding is necessary: pre-train one deep encoder to read text in both directions at once, then bolt on a single output layer and fine-tune the whole thing. The catch was that "both directions at once" had been considered cheating — a bidirectional model can trivially see the word it is predicting.

Key Findings
  • The masked-language-modeling objective is the unlock. By hiding ~15% of tokens and predicting them from full left-and-right context, BERT learns deep bidirectional representations without the model leaking the answer to itself — the thing that had kept prior work left-to-right.
  • One architecture, eleven tasks. The same pre-trained weights plus a thin task head set new state-of-the-art on GLUE, SQuAD v1.1/v2.0, and SWAG, replacing the per-task encoders that dominated the field.
  • Scale pays off cleanly. BERT-large (340M params) beats BERT-base on every task, including small datasets — early evidence that pre-training scale, not task-specific cleverness, was the lever.
  • Next-sentence prediction adds the sentence-pair reasoning that QA and entailment need, though later work (RoBERTa) showed this objective was the weakest link.
Methodology

The two-stage recipe — expensive unsupervised pre-training once, cheap supervised fine-tuning many times — is the template most of the field still follows. It is also where BERT diverges from GPT: GPT's autoregressive, left-to-right objective makes it a natural text generator, while BERT's masked, bidirectional encoder is built for understanding, not generation, and cannot generate fluently.

Who It's For

Great fit if you are building classifiers, extractive QA, NER, or any task where you encode text and read out a label, and you want a baseline that still beats most from-scratch models. Look elsewhere if you need open-ended generation (use a decoder/GPT-style model), or if you need a current production system — DeBERTa, RoBERTa, and modern encoders have since surpassed the original weights. Read it as the paper that made "pre-train then fine-tune" the default, and as the encoder counterpoint to the GPT generation lineage.

Back

Information

  • Websitearxiv.org
  • OrganizationsGoogle AI Language
  • AuthorsJacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
  • Published date2018/10/10

Categories

Tags

More Items

Large Language Model Papers2026
Icon for item

TPPO: Tracking-Value PPO For Language Model Reasoning

Futing Wang, Chenxi Li +7Zhejiang University, Westlake University +5

Improves token-level credit assignment for RL fine-tuning of language models by parameterizing the critic as cumulative, position-wise log-odds increments (Tracking-Value PPO), enabling more accurate value-evolution tracking under sparse terminal rewards and boosting math-reasoning and coding benchmarks.

rlLLMreasoningqwenmath+3
Large Language Model Papers2025
Icon for item

Foundations of Large Language Models

Tong Xiao, Jingbo ZhuNLP Lab, Northeastern University, NiuTrans Research

A concise textbook-style book that explains foundational concepts and techniques for large language models, covering pre-training, generative models, prompting, alignment, inference, and reasoning. Structured as self-contained chapters for readers with some ML/NLP background or those seeking a principled introduction to LLM foundations.

foundation-modelLLMNLPpaperbook+5
Large Language Model Papers2026
Icon for item

TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models

Xin Wang, Hao Yu +10Affiliation: Alibaba Token Hub, Alibaba Group, Affiliation: Ohio State University

Proposes TRACE, an FP4 quantization framework for RL of MoE LLMs that uses rollout-side FP4 outcomes to guide training-side rounding and caches mantissa/scale from deeper layers to limit overhead. Preserves BF16-level RL performance while enabling up to 5.4× rollout speedup.

fp4quantizationmoeRLLLM+4