Everyone remembers attention from the 2017 Transformer paper, but the mechanism was born here, three years earlier, solving a humbler problem: a single fixed-length vector simply could not hold a long sentence. The real insight wasn't a new network — it was reframing translation as a search. Instead of forcing the encoder to compress everything into one bottleneck vector, the decoder learns to look back and weight the source words that matter for the word it's about to produce. That reframing, not the BLEU score, is why this paper still gets cited.
Key Findings
- The fixed-length vector was the bottleneck. Classic seq2seq encoders squeezed an entire source sentence into one vector; quality collapsed as sentences grew longer. Naming that failure mode is half of what made the fix obvious.
- Soft alignment, learned end-to-end. Rather than hard-segmenting the source, the model computes a probability distribution over all source positions for each target word — differentiable, so it trains jointly with the rest of the network. This is attention before it had the name.
- It held its own against phrase-based SMT. On English-to-French, the attention model reached performance comparable to the heavily hand-engineered statistical systems of the day, without their feature engineering — and stayed robust on the long sentences where plain encoder–decoders broke down.
- The alignment maps were interpretable. Visualizing the attention weights showed linguistically sensible word correspondences, an early hint that attention was learning structure, not just smoothing the bottleneck away.
How It Works
The encoder is a bidirectional RNN producing one annotation per source word; the decoder, at each step, scores its current state against every annotation, normalizes those scores into weights, and reads a context vector that is their weighted sum. Order is preserved by the RNN itself — there are no positional encodings here, because recurrence still does that job.
Why It Still Matters / When to Skip
Great fit if you want to see where attention actually started and to read the cleaner, RNN-grounded version of the idea before the Transformer abstracted recurrence away entirely. Look elsewhere if you want a current architecture to build on — modern systems dropped the RNN backbone and use self-attention throughout; this is the conceptual origin, not production code.
