One mechanism — no recurrence, no convolutions.
Before 2017, the best translation models read a sentence one word at a time, each step waiting on the one before it. That sequential habit was powerful but costly: it could not be parallelized, and the link between two distant words had to survive a long relay, fading with every hop.
The authors made one bet — throw all of that out and rely solely on attention, letting every word look directly at every other word in a single parallel step. The payoff is two-fold: training parallelizes across all positions at once, and the path between any two words collapses to a single hop, so long-range dependencies become easy to learn.
Attention alone — applied in parallel, to every pair of words at once — turned out to be all you need.
The machine from orbit
What is this thing, and what shape is it?
Step 1 · The black box
A box that translates — and we get to open it.
Here is the whole story in one picture. An English sentence goes in, a German sentence comes out. This exact sentence — the paper's headline task was WMT 2014 English→German translation — will follow us through all twelve steps:
In 2017 this box out-translated every system built in the preceding years — including ensembles of models — after just 3.5 days of training on eight GPUs. The rest of this page opens the box, one layer of wrapping at a time.
Why did it win? The models before it read text the way you read a ticker tape: one word at a time, each step waiting on the last — so nothing could be computed in parallel, and a link between two far-apart words had to survive every step in between, fading as it went.
From the abstract: “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”
Step 2 · Opening the box
Three zooms, and the box becomes a floor plan.
You've seen the box translate our sentence. Now we unwrap it in three zooms — each picture is the same machine, drawn with one more level of detail.
The box is just two towers: a reading tower and a writing tower. Each tower is six identical floors. And each floor does only two (or three) things. That's the whole inventory.
Zoom 1 · two towers
The box is just two towers: one reads the English, one writes the German.
Zoom 2 · six floors each
Each tower is six identical floors (N = 6). The reading tower's final output is handed to every floor of the writing tower.
Zoom 3 · inside one floor
Dissect one floor: an encoder floor does only [self-attention → feed-forward]; a decoder floor adds one block in between.
Every word travels its own lane up through the tower. The only place where lanes interact is self-attention — the feed-forward block touches each word separately. That's why all positions can be processed at once, and why training parallelizes.
| Layer type | Complexity | Sequential ops | Max path length |
|---|---|---|---|
| Recurrent | O(n·d²) | O(n) | O(n) |
| Convolutional | O(k·n·d²) | O(1) | O(logk n) |
| Self-attention | O(n²·d) | O(1) | O(1) |
Step 3 · Words become vectors
Before anything happens, words become lists of numbers.
You know the floor plan. Before our sentence can enter the bottom floor, there's a conversion step: neural networks compute with numbers, not letters.
Each word is swapped for its embedding — a learned list of 512 numbers that encodes the word's meaning, where words used similarly end up with similar lists. So “I am a student” enters the machine as four strips of 512 numbers each:
One detail that makes the whole tower work: only the bottom floor sees raw embeddings. Every floor above consumes the floor below's output — four vectors in, four vectors out, always the same shape. Same shape in, same shape out is exactly what lets you stack six identical floors.
Two ingredients are still missing. Nothing here records word order (we fix that in Step 8), and at the far end we'll need to turn a vector back into a word (Step 11). Park both questions — first, the heart of the machine.
Self-attention, three passes
Feel it · compute it by hand · compress it into one line
Step 4 · Pass 1 — feel it
When the machine reads “its”, what does “its” mean?
Our four words are now four vectors riding their own lanes. Time for the one block where lanes interact — and before any math, you should feel what it does.
Take a sentence from the paper's own appendix: “The Law will
never be perfect, but its application should be just …”. When
the machine reads its — possessive of what,
exactly? A useful representation of “its” must contain some “Law”.
That is self-attention: while encoding a word, the model scores every other word for relevance and blends the useful ones into this word's representation. Here, “its” pulls in “Law” and “application”. Try it:
Hover / tap an underlined word
Arc thickness = attention weight. The paper's appendix shows two trained heads attending sharply from its to Law and to application — resolving the pronoun with no grammar rules provided.
Second exhibit · a nine-word reach
Also from the appendix: in encoder layer 5 of 6, heads attend from making across nine words to more and difficult — completing the phrase “making … more difficult”.
Arc weights here are illustrative; the qualitative patterns are taken from the paper's appendix visualizations of real trained heads.
Step 5 · Pass 2 — compute it by hand
Six small steps. Real numbers. No magic left.
You've felt what attention does. Now do it yourself, with actual numbers, so there's nothing left to take on faith.
We shrink the sentence to two words — Attention works —
and follow word #1 (“Attention”) through one attention pass, at the
real model's dimensions. Six rows; each one is a single arithmetic
move.
Make a query, a key, and a value for each word
Multiply each word's 512-number embedding by three learned matrices — WQ, WK, WV — to get three 64-number vectors. The query asks “what am I looking for?”, the key advertises “what do I contain?”, and the value is “what I hand over if you pick me.”
Score: dot word #1's query against every key
A dot product multiplies two vectors position by position and adds it all up — a big result means the directions align. It's our relevance meter: how well does each key answer q₁'s question?
Scale: divide by 8
8 = √64, the square root of the key length. With 64 positions summed, dot products grow large — and huge scores park the next step (softmax) in a flat region where learning gradients die. Dividing by √dk keeps the numbers in the healthy zone.
Softmax: turn scores into weights
Softmax squashes any list of scores into positive weights that sum to 1 — by exponentiating each score and dividing by the total. Note what it did to our gap: a 12-vs-10 score lead became an 88-vs-12 weight lead.
Weigh each value by its weight
Multiply every word's value vector by its weight. The losing word isn't erased — it's shrunk toward zero, still faintly present.
Sum — and you have z₁
Add the weighted values. The result z₁ is the new representation of “Attention”: mostly its own content, plus a measured dose of “works”.
Drag the scores — watch row 4 happen live
Softmax exaggerates gaps: a small score lead becomes a big weight lead. Push the scores equal and the weights split 50/50; open a gap of 4+ and the winner takes nearly everything.
A token mostly keeps itself, but measurably blends in what it needs from others. That blend — recomputed fresh for every word, in every layer — is the entire trick.
Checking row 4: e¹²/(e¹² + e¹⁰) = 1/(1 + e⁻²) ≈ 0.88. The same pass runs for word #2 with q₂ — every word gets its own z.
Step 6 · Pass 3 — compress it
Your six steps, folded into one line.
You just did six steps for one word. Stack every word's q, k and v into three matrices Q, K, V — one row per word — and all six steps for all words collapse into one line:
Attention(Q, K, V) = softmax(QKᵀ√dk)V
QKᵀ = rows 2–3 · ÷√dk = row 3 · softmax = row 4 · ·V = rows 5–6
This formula is not new information — it's a receipt for work you
already did by hand. The only addition is bookkeeping:
QKᵀ dots every query against every key
simultaneously, producing the all-pairs score table.
See the same six steps in matrix form
row 1 → Q = XWQ, K = XWK, V = XWV
rows 2–3 → S = QKᵀ / √dk (every score, scaled)
row 4 → A = softmax(S) (each row sums to 1)
rows 5–6 → Z = AV (each row of Z is one word's z)
Real dimensions: X is n × 512; WQ, WK, WV are 512 × 64, so Q, K, V are n × 64 and the score table is n × n.
Step 7 · Multi-head attention
The real model runs eight of these at once.
You now know how one head computes attention. The real model runs eight of them at once — here's why.
Look back at Step 4: “its” needed both “Law” and
“application”. A single head has one weight budget that sums to 1,
so it must split — averaging the two relationships into a muddier
blend of both. The paper's fix: run h = 8 independent
attention units in parallel, each with its own learned
WQ, WK, WV. Each head can develop
its own habit — one tracks pronouns, another tracks word order —
then the eight results are concatenated and projected back to 512
numbers by one more learned matrix, WO.
Pick a head, watch the pattern switch
Looks at itself
Each position attends mostly to its own token — keeping identity intact.
Habits illustrative — the appendix shows real heads specializing in syntax-like and semantics-like roles. With one head, averaging inhibits this.
From the ablations: a single head is 0.9 BLEU worse than the 8-head base; 32 heads is also worse than 8. More habits help — up to a point.
Step 8 · Positional encoding
Order, restored with sine waves.
Time to pay the debt from Step 3. Nothing you've computed so far cares about word order — shuffle “I am a student” into “student a am I” and every dot product in Step 5 comes out identical, because attention compares contents, not positions.
The fix is almost embarrassingly simple: before the first floor, add a position-dependent vector to each word's embedding — same length, 512 numbers — so that “am, the 2nd word” enters the tower as a slightly different vector than “am, the 5th word” would. The paper builds these vectors from sines and cosines of different wavelengths. Here's a toy version with d_model = 4, interleaving sin and cos exactly like the paper:
sin(pos/1) dim 1
cos(pos/1) dim 2
sin(pos/100) dim 3
cos(pos/100) 0 · I 0.00 1.00 0.00 1.00 1 · am 0.84 0.54 0.01 1.00 2 · a 0.91 −0.42 0.02 1.00 3 · student 0.14 −0.99 0.03 1.00
Read it column by column. Dims 0–1 spin fast — they change completely between neighboring words. Dims 2–3 barely move — they encode coarse position. The real model has 512 dims spanning wavelengths from 2π up to 10000·2π.
Why sines instead of just learning a vector per position? Two nice properties. For any fixed offset k, PE(pos+k) is a linear function of PE(pos) — so “three words to the left” is an easy, learnable relationship. And waves keep going: sinusoids may extrapolate to sentences longer than any seen in training. Learned position embeddings scored nearly identically in the paper's tests; the authors chose sinusoids for that extrapolation chance.
Step 9 · Residuals & LayerNorm
The plumbing that makes six floors trainable.
The reading tower is nearly assembled: embeddings + positions enter, self-attention mixes, feed-forward refines. Two last fittings keep a 12-floor building from collapsing during training.
First, the residual connection: each sub-layer's input is added back to its output — a highway that lets the original signal (and, during training, the gradients) skip past any layer that isn't helping yet. Second, LayerNorm: a re-balancing step that rescales the 512 numbers to a standard spread, so values don't drift out of range as they climb floor after floor.
LayerNorm(x + Sublayer(x))
this wrapper appears around every sub-layer, in all 12 floors
The feed-forward block itself: FFN(x) = max(0, xW₁ + b₁)W₂ + b₂ with d_ff = 2048 — applied to each position identically and independently. All sub-layer outputs are d_model = 512, which is what keeps the floors stackable.
The machine writes
Decoding, choosing words, and learning to be right
Step 10 · The decoder
It writes one word at a time — and may never peek ahead.
The reading tower is built. Now the writing tower — the decoder — turns its work into German.
The encoder runs once over “I am a student” and hands its final K and V to every decoder floor. There, cross-attention works exactly like the attention you computed in Step 5 — except Q comes from the decoder (“what do I need right now?”) while K and V come from the encoder (“here's the source sentence”).
Generation is a loop: emit a word, feed it back in as input, emit
the next — until the model produces ⟨eos⟩, a special
end-of-sentence token that means “I'm done.” Run the loop yourself:
Step the decoder — watch the German emerge
Source — encoded once, attended at every step
Target — written so far
Step 1 of 5 — top-3 candidates for the first word.
Arc = cross-attention into the current step. Notice the decoder leaning on “I” while writing ich, on “am” while writing bin — the source stays available the whole time. Probabilities are toy values for our running example.
One rule makes this loop honest: masking. While writing word 3, the decoder's self-attention may only look at words 1–2. That sounds obvious during generation — the future doesn't exist yet — but during training the whole correct German sentence is sitting right there in the input. Masking (setting future scores to −∞ so softmax gives them weight 0) is what stops the model from simply copying the answer instead of learning to predict it.
Three attentions, one mechanism: encoder self-attention (Step 4), decoder masked self-attention (this mask), and encoder→decoder cross-attention (Q from decoder; K, V from encoder output).
Step 11 · The output head
From a 512-number vector to the word “bin”.
The stepper showed words appearing with probabilities attached. Here's where those probabilities come from — the second debt from Step 3.
The decoder's top floor outputs one 512-number vector per position. A final Linear layer — a learned scorer — assigns every word in the vocabulary a raw score, called a logit. Softmax (the same one from Step 5) turns those scores into probabilities. Then you pick.
Watch it on a toy six-word German vocabulary at the moment the model should say “bin”. The training target is a one-hot vector — all zeros except a single 1 on the correct word:
Untrained modelrandom junk — ≈ uniform
Trained modelafter 4.5M sentence pairs
The targetone-hot for “bin”
Training is the journey from the left column to the middle one. The loss function — cross-entropy — measures how far the predicted bars are from the one-hot bars; every weight update nudges the matrices of Steps 5–9 so the right bar grows a little taller. Repeat across 4.5 million sentence pairs.
One last decision remains at inference time: how do you pick? Greedy decoding always takes the tallest bar — fast, but a single early misstep can't be undone. Beam search keeps several candidate sentences in flight:
Two paper details worth knowing: the final Linear layer shares its weights with the embedding layers (scaled by √d_model). And training used label smoothing 0.1 — the model is taught to never be 100% sure, which worsens perplexity (≈ how surprised the model is by held-out text) but improves BLEU (≈ overlap with reference translations, the field's yardstick).
Step 12 · Results & recap
Did it work? — and can you now explain it?
The machine is fully assembled: it reads (Steps 2–9), writes (Step 10), and chooses words (Step 11). Final question: was attention really all you need?
Report card · WMT 2014 newstest (BLEU)
Transformer (big) · EN→FR
prior best ensemble · EN→FR
Transformer (big) · EN→DE
prior best ensemble · EN→DE
0BLEU →45
Now you can explain it. Six questions — answer each out loud before opening it. If all six come easily, you've genuinely got this paper.
Why divide by √d_k?
Dot products over 64 dimensions grow large (their variance grows with d_k), and big scores park softmax in a flat region where gradients vanish. Dividing by √d_k = 8 keeps learning alive — you did it yourself in Step 5, row 3.
Why eight heads instead of one?
One head's weights sum to 1, so attending to “Law” and “application” at once means muddying both. Eight independent heads learn eight habits. Ablations: 1 head is 0.9 BLEU worse; 32 heads is also worse than 8.
Why add positional encodings?
Attention compares contents, not positions — shuffle the words and every score is identical. Adding sinusoid vectors makes position part of the content; PE(pos+k) being linear in PE(pos) makes relative offsets easy to learn.
Why did training parallelize so well?
Every word rides its own lane, and the only cross-lane step — attention — is one big matrix multiply over all positions at once: O(1) sequential operations, versus O(n) for a recurrent chain that must go word by word.
What may the decoder see that the encoder may not?
Trick question — it's the reverse. The encoder sees the whole source sentence, both directions. The decoder may not see future target words: the mask sets their scores to −∞ so it must predict, not copy.
How does a 512-number vector become a word?
A final Linear layer scores every vocabulary word (logits), softmax turns scores into probabilities, and decoding picks — greedily, or with a beam of candidate sentences (the paper used beam 4, α = 0.6).
What happened next — BERT (2018) is this paper's encoder stack; GPT is its decoder stack. The same block now reads images, audio, code and protein sequences. Every “transformer” in every model card traces back to the twelve steps you just walked.
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.” — the abstract, Vaswani et al., 2017
Nearly every modern model is a Transformer.
The architecture didn't stay in translation. Within a year its two halves had split into the two families that now dominate AI — and the same block spread far beyond text.
BERT
Took the encoder stack and pre-trained it to read in both directions at once — reshaping how machines understand language.
GPT
Took the decoder stack and scaled it up, again and again — the lineage behind today's chat assistants and large language models.
Vision · audio · code
The same attention block now powers image, speech, protein and multimodal models far past the translation task it was built for.
Every “transformer” in every model card traces back to this one 2017 paper.