AIAny
Natural Language Processing Papers2014

Neural Machine Translation by Jointly Learning to Align and Translate

First model to make a decoder dynamically focus on different source words instead of cramming a whole sentence into one fixed vector — the soft-alignment idea that became "attention" and, three years later, powered the Transformer.

Visual ExplainerVisit Website

Introduction

Everyone remembers attention from the 2017 Transformer paper, but the mechanism was born here, three years earlier, solving a humbler problem: a single fixed-length vector simply could not hold a long sentence. The real insight wasn't a new network — it was reframing translation as a search. Instead of forcing the encoder to compress everything into one bottleneck vector, the decoder learns to look back and weight the source words that matter for the word it's about to produce. That reframing, not the BLEU score, is why this paper still gets cited.

Key Findings
  • The fixed-length vector was the bottleneck. Classic seq2seq encoders squeezed an entire source sentence into one vector; quality collapsed as sentences grew longer. Naming that failure mode is half of what made the fix obvious.
  • Soft alignment, learned end-to-end. Rather than hard-segmenting the source, the model computes a probability distribution over all source positions for each target word — differentiable, so it trains jointly with the rest of the network. This is attention before it had the name.
  • It held its own against phrase-based SMT. On English-to-French, the attention model reached performance comparable to the heavily hand-engineered statistical systems of the day, without their feature engineering — and stayed robust on the long sentences where plain encoder–decoders broke down.
  • The alignment maps were interpretable. Visualizing the attention weights showed linguistically sensible word correspondences, an early hint that attention was learning structure, not just smoothing the bottleneck away.
How It Works

The encoder is a bidirectional RNN producing one annotation per source word; the decoder, at each step, scores its current state against every annotation, normalizes those scores into weights, and reads a context vector that is their weighted sum. Order is preserved by the RNN itself — there are no positional encodings here, because recurrence still does that job.

Why It Still Matters / When to Skip

Great fit if you want to see where attention actually started and to read the cleaner, RNN-grounded version of the idea before the Transformer abstracted recurrence away entirely. Look elsewhere if you want a current architecture to build on — modern systems dropped the RNN backbone and use self-attention throughout; this is the conceptual origin, not production code.

Back

Information

  • Websitear5iv.labs.arxiv.org
  • OrganizationsUniversité de Montréal, Jacobs University Bremen
  • AuthorsDzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
  • Published date2014/09/01

Categories

Tags

More Items

Reinforcement Learning Papers2026
Icon for item

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Bingxi Hou, Guochao Jiang +5Guohua Liu, Yuewei Zhang +1

Uses an intermediate teacher checkpoint to gate transfer of RL-induced policy shifts in on-policy distillation by checking directional agreement between early and late teacher changes; suppresses endpoint shifts that reverse and requires one extra frozen-teacher scoring pass.

distillationrlqwenreasoningcode+2
Large Language Model Papers2026
Icon for item

TPPO: Tracking-Value PPO For Language Model Reasoning

Futing Wang, Chenxi Li +7Zhejiang University, Westlake University +5

Improves token-level credit assignment for RL fine-tuning of language models by parameterizing the critic as cumulative, position-wise log-odds increments (Tracking-Value PPO), enabling more accurate value-evolution tracking under sparse terminal rewards and boosting math-reasoning and coding benchmarks.

rlLLMreasoningqwenmath+3
Large Language Model Papers2025
Icon for item

Foundations of Large Language Models

Tong Xiao, Jingbo ZhuNLP Lab, Northeastern University, NiuTrans Research

A concise textbook-style book that explains foundational concepts and techniques for large language models, covering pre-training, generative models, prompting, alignment, inference, and reasoning. Structured as self-contained chapters for readers with some ML/NLP background or those seeking a principled introduction to LLM foundations.

foundation-modelLLMNLPpaperbook+5