Don't memorize the sentence — look back at it.
The translation models of 2014 read a whole source sentence, squeezed it into one fixed-length vector, and asked the decoder to write the translation from that single summary. It worked for short sentences and fell apart on long ones: no fixed-size vector can hold an arbitrarily long sentence without losing detail.
This paper made one bet — stop forcing everything through that bottleneck. Keep one vector per source word, and let the decoder, at each step, soft-search the source for the words that matter to the word it is about to produce. Translation becomes a search, not a recall test. That reframing — not the BLEU score — is why the paper is still cited.
Reading a long sentence and writing it down from memory is hard. Reading it while you translate, and glancing back as needed, is easy.
The machine from orbit
What is this thing, what broke before it, and what shape is it?
Step 1 · The black box
A box that translates — and we get to open it.
Here is the whole story in one picture. A French sentence goes in, an English sentence comes out. We will use one real example from the paper's own alignment figure all the way through — a short slice of it to start:
“Neural machine translation” was new in 2014: instead of a pipeline of hand-tuned pieces, build one network that reads the source and writes the target, and train the whole thing end to end. The dominant recipe was the encoder–decoder (also called seq2seq): an encoder RNN reads the source word by word, and a decoder RNN writes the target word by word.
The rest of this page opens this box. But first we have to see exactly what was broken inside the version that came before it — because the fix is obvious only once you can name the failure.
From the abstract: “we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder–decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence.”
Step 2 · The bottleneck
One vector cannot hold a whole sentence.
You've seen the box translate our phrase. Now look at how the previous design tried to do it — and where it choked.
In the plain encoder–decoder, the encoder reads every source word and
keeps updating a single running state. After the last word, that one
final state — call it c, the context vector — is
the entire summary handed to the decoder. Every English word
must be written from c alone; the source words
themselves are gone.
A fixed-size c is fine for a 5-word sentence and
disastrous for a 40-word one: the longer the source, the more detail
you have to cram into the same number of slots. Cho et al. (2014) had
already shown plain encoder–decoder quality dropping sharply as
sentences grow. The bottleneck is structural, not a tuning problem.
So the question this paper asks is narrow and sharp: what if the decoder didn't have to work from one frozen summary? What if, at each output word, it could reach back into the source and read the parts it needs right now?
Step 3 · The floor plan
Keep one vector per word — and read both directions.
You've named the bottleneck: one frozen c. The fix starts
with a different encoder — one that never throws the per-word
information away.
Instead of keeping only the final state, the new encoder emits
one vector per source word. The paper calls each one an
annotation hj — a
small summary of the source centred on word j. To make each
annotation aware of both the words before and after it, the
encoder is a bidirectional RNN: a forward pass reads left to
right, a backward pass reads right to left, and the two halves are
glued together.
Now the decoder has a shelf of annotations — h₁, h₂, h₃, h₄ —
one per source word, instead of a single squashed c.
Word order is still handled by the RNN itself, so there are no
positional codes here. The only thing left to invent is the rule the
decoder uses to pick which annotations to read at each step. That rule
is the heart of the paper, and the whole of Part B.
The alignment mechanism
Feel it · compute one context vector by hand · compress it into three lines
Step 4 · Pass 1 — feel it
For each English word, which French words matter?
The decoder now has a shelf of annotations, one per source word. Before any math, feel what it does with them: as it writes each English word, it weights the French words by how relevant they are right now.
This is the paper's own example. The grid below is an alignment heatmap: one row per English word it produces, one column per French word. A bright cell means “while writing this English word, the decoder leaned heavily on that French word.” Step through the output and watch the bright cell move:
Step through the output — watch the bright row light up
Bright = high attention weight. Most of the map hugs the diagonal — English and French run roughly in parallel. But watch rows 4–6: Economic · European · Area line up against économique · européenne · zone in reversed order. The decoder jumped over two words and looked back — the soft-alignment handling a real adjective–noun reordering.
Heatmap values here are illustrative, but the pattern — near-diagonal alignment plus the reversed “European Economic Area → zone économique européenne” block — is taken straight from Figure 3 of the paper.
Step 5 · Pass 2 — compute one context vector by hand
Score, soften, blend. Real numbers. No magic left.
You've felt the decoder pick which French words to read. Now do it yourself, with actual numbers, for a single decoder step — so there's nothing left to take on faith.
Setup: the decoder is about to produce one English word. Its current
state is s. The source has three annotations,
h₁ h₂ h₃ (for la, zone, économique).
We will build the one context vector c the decoder reads
this step, in four moves.
Score: ask “how well does each source word fit right now?”
For each annotation hj, a tiny shared network reads
the decoder state s together with hj and
returns one number — the alignment energy
ej. Bigger means “more relevant to the
word I'm about to write.” (We'll see that little network in Step
7.)
Soften: turn energies into weights that sum to 1
Softmax exponentiates each energy and divides by the total, so the three numbers become positive weights that add to 1 — a genuine probability distribution over the source words. The weights αj are exactly the brightness in the heatmap.
Weigh each annotation by its weight
Scale every annotation vector by its α. The losing words aren't
deleted — they're shrunk toward zero, still faintly present. Here
h₂ (“zone”) keeps 84% of its strength; the others
fade.
Sum — and you have the context vector c
Add the weighted annotations. The result c is a
custom-blended source summary, built fresh for this one output
step: mostly “zone”, with a touch of “la” and “économique”.
Next step the decoder recomputes it from scratch with a new α.
Drag the energies — watch the weights and the blend move
Softmax exaggerates gaps. Make the three energies equal and the weights split evenly — c becomes a flat average of the whole source (the bottleneck is back). Open a clear lead and one source word dominates c — that's a sharp, confident alignment.
The decoder reads a different context vector at every output step — each one a freshly weighted blend of the same source annotations. That per-step blend, learned end to end, is the entire mechanism the world later called attention.
Checking step 2: e²/(e²+e⁴+e¹) ≈ 0.11, e⁴/(…) ≈ 0.84, e¹/(…) ≈ 0.05. Energies and annotations here are toy values; the procedure is exactly the paper's Equations 5–6.
Step 6 · Pass 3 — compress it
Your four moves, folded into three lines.
You just built one context vector by hand. Here are the same
four moves as the paper writes them — three short equations, for
output position i over all source positions
j:
eij = a(si−1, hj) · αij = exp(eij)Σk exp(eik) · ci = Σj αij hj
score = step 1 · α = step 2 (softmax) · weighted sum = steps 3–4
No new information — this is a receipt for the work you already did.
eij is the energy of source word j for output
word i; αij is its softmax weight; and
ci is the blended context the decoder reads at
step i. The one thing to carry forward: that subscript
i on c.
The plain encoder–decoder used one context vector
c for the entire translation. This model uses a
distinct ci for each target word.
That single subscript is the bottleneck dissolving: the decoder is no
longer translating from a frozen summary, but re-reading the source,
its own way, at every step.
How the decoder uses ci
si = f(si−1, yi−1, ci) — the decoder's next state depends on its last state, the last word it wrote, and this step's context.
p(yi | y<i, x) = g(yi−1, si, ci) — the next-word probability.
The decoder and the alignment network are jointly trained: because softmax and the weighted sum are differentiable, gradients flow back through the α's, so the model learns to align as a by-product of learning to translate — no alignment labels are ever provided.
Step 7 · The alignment network
The scorer is one tiny tanh layer.
One gap remains: what is the scoring function
a(s, h) from Step 5? It's the smallest possible neural
network, and it is the only truly new trainable piece the paper adds.
The alignment model a takes the decoder state
si−1 and one annotation hj,
runs them through a single hidden layer with a tanh
squash, and reads off one number:
eij = va⊤ tanh(Wa si−1+Ua hj)
Wa, Ua project into a hidden layer (n′ = 1000) · va collapses it to one score
Cost note: this little scorer runs once for every (output word, source word) pair, so its work grows with sentence length times target length — the price of letting the decoder look anywhere it likes.
Does it work — and what came of it?
Step the decoder · read the scoreboard · explain it back
Step 8 · The whole loop, in motion
Write a word, re-aim, write the next.
You have all the pieces: annotations, energies, softmax weights, a per-step context vector. Now watch them run as a loop. Step the decoder and see the alignment re-aim before each English word.
Step or play — the arcs show where cᵢ is reading from
Source annotations (French)
Decoder output (English)
The arcs are the α weights. For most words one French source lights up — a clean diagonal. On “European Economic Area” the arcs cross: the decoder reaches back to zone for “Area”, then forward again — exactly the reordering you saw in the heatmap.
Step 9 · The scoreboard
Bigger gains on long sentences — exactly where the old model broke.
You've seen the mechanism run. Did it actually translate better? Two numbers tell the whole story: BLEU, and what happens as sentences get long.
BLEU is the standard translation score
(higher is better, 0–100). The paper trains two model families to a
30-word or 50-word limit: RNNencdec (the old fixed-vector
model) and RNNsearch (this one). Same data, same size —
only the alignment mechanism differs.
BLEU on the test set — all sentences
old · 30-word limit
this model · 30-word limit
old · 50-word limit
this model · 50-word limit
phrase-based SMT baseline
0BLEU →33
Step 10 · Now you can explain it
Six questions. Answer each out loud first.
You walked the whole machine — bottleneck, annotations, energies, softmax, per-step context. If these six come easily, you've genuinely got this paper.
What exactly was the “bottleneck”?
The plain encoder–decoder squeezes the whole source into one fixed-length vector c, and the decoder writes everything from it. On long sentences there's no room — detail is lost, BLEU collapses. This paper keeps one annotation per word instead.
What is an annotation h_j?
One vector per source word, produced by a bidirectional RNN: h_j = [forward state ; backward state]. It summarizes the whole source but is centred on word j, so it carries context from both sides.
How is the context vector c_i built?
Score every annotation against the decoder state (e_ij), softmax the scores into weights that sum to 1 (α_ij), then take the weighted sum of annotations: c_i = Σ_j α_ij h_j. You did this by hand in Step 5.
What's different from the old model, in one symbol?
The subscript i on c. The old model used one c for the whole sentence; this one uses a distinct c_i for every target word — the decoder re-reads the source at every step instead of working from a frozen summary.
Who labeled the alignments?
Nobody. Softmax and the weighted sum are differentiable, so the α's are trained jointly with the rest of the network. The model learns to align as a side effect of learning to translate — no alignment supervision is ever given.
Why does it survive long sentences?
It never compresses the sentence into one fixed vector. At each step it reads only the few source words it needs. Figure 2: BLEU stays flat out to 50+ words where the fixed-vector model falls off a cliff.
What happened next — in 2017 “Attention Is All You Need” kept this soft-search idea, threw out the RNN that carried it, and let every word attend to every other word in parallel. The alignment weights you just computed became the self-attention at the core of BERT, GPT and nearly every model since.
“We achieve a translation performance comparable to the existing state-of-the-art phrase-based system… qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.” — the abstract, Bahdanau, Cho & Bengio, 2014
The mechanism that ate the field.
This paper introduced soft attention to fix a translation bottleneck. The bottleneck got fixed — but the mechanism turned out to matter far more than the fix.
“Attention”
Luong et al. and others refined the scoring and made it standard equipment in sequence-to-sequence models — and gave the mechanism its lasting name.
The Transformer
“Attention Is All You Need” kept the soft-search idea, dropped the RNN entirely, and ran attention everywhere in parallel — the architecture behind modern AI.
BERT · GPT · beyond
Every large language model, and most vision, speech and protein models, are built on the attention this paper first made differentiable and learnable.
Every “attention” in every model card traces back to this one 2014 idea: don't memorize the input — learn where to look.