Give the network a memory tape — and make every read and write differentiable.
A plain neural network has nowhere to put intermediate results. It can recognize a cat, but it cannot reliably follow a procedure — copy this list, then sort it — because it has no scratchpad to hold state across steps. A 1950s computer solved this with a separate memory and a controller that reads and writes addresses. But “read address 7” is a hard, discrete jump: you cannot take its gradient, so you cannot learn it by backprop.
The bet of this paper: take that textbook computer diagram — a controller plus an addressable memory — and make every part of it smooth. Instead of reading one address, a head reads a weighted blend of all memory rows, with weights that sum to one. Now “which row” is a soft distribution, the whole loop is differentiable, and the read/write logic can be learned by gradient descent.
A learned computer, not a fixed circuit — and that soft weighting over memory rows is exactly the attention you already know.
The machine from orbit
What is this thing, and what shape is it?
Step 1 · The black box
A box that watches examples — and learns to copy.
Here is the whole story in one picture. We feed in a short sequence of
bit-vectors, then a special END marker. The box must then
replay the same sequence back, in order. Nobody ever tells it
how — it sees only inputs paired with the correct outputs, and
has to figure out the procedure. This exact task — the paper's headline
copy task — will follow us through all ten steps:
Copying looks trivial — until you ask a network to do it. A plain recurrent network has to squeeze the whole input into one fixed-size hidden state and then unspool it; the longer the sequence, the more it blurs. The NTM instead writes each vector onto a memory tape, then walks back along the tape and reads them out. That is why, trained only on lengths up to 20, it later copies length-120 sequences nearly perfectly — it learned the procedure, not the training lengths.
The rest of this page opens the box. First the floor plan (Part A), then the one mechanism that makes it all differentiable (Part B), then the evidence that it actually learned to compute (Part C).
From the abstract: “We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine but is differentiable end-to-end.”
Step 2 · The floor plan
Two parts only: a controller, and a memory matrix it talks to.
You've seen the box copy our sequence. Now we open it — and inside is the oldest diagram in computing, redrawn so it can be trained.
The NTM has exactly two pieces. The controller is an ordinary
neural network (a feedforward net, or an LSTM —
a recurrent net that carries a memory of its own across steps). The
memory is just a matrix M: N rows, each a
vector of length M. In the copy experiments that matrix is
128 × 20 — 128 slots, each holding 20 numbers.
The controller never touches memory directly. It works through heads — a read head and a write head — the way a tape machine works through a read/write head hovering over a tape. Each step, the controller emits some numbers; the heads turn those numbers into an access pattern over the rows.
Separating computation (controller) from storage (memory matrix) means you can grow capacity by adding rows — without ballooning the parameter count. The same lesson recurs in every later memory-augmented and retrieval system.
Step 3 · The soft trick
Don't pick a row. Take a weighted blend of all of them.
The floor plan has a controller, a memory matrix, and heads. But how does a head “point” at a row without making a hard, ungradable jump? This is the whole trick of the paper.
A real computer reads one address: row 7, full stop. That choice is discrete — nudge the controller's numbers a hair and the chosen row either flips to 8 or doesn't. There is no slope to descend, so backprop is stuck.
The NTM replaces “pick row 7” with a weighting
w — one non-negative number per row, summing to 1. A
weighting of (0,0,1,0,…) means “all of row 3”; a
weighting of (0,0,0.6,0.4,…) means “mostly row 3, a bit
of row 4”. Reading returns the weighted average of the rows;
writing spreads its edit across them by the same weights. Because the
weighting moves smoothly as the controller's numbers change, the whole
machine is now differentiable.
A weighting that sums to 1, used to take a weighted average over a set of stored vectors, is exactly attention. Three years before “Attention Is All You Need”, the NTM was already doing soft, differentiable addressing — over an explicit memory tape rather than over the input tokens.
The one mechanism, hand-calculated
Reading, writing, and how a head decides where to point.
Step 4 · Reading
A read is just the weighting times the rows — a weighted average.
We have a weighting w that sums to 1. The simplest thing a
head does with it is read — so let's compute one read by hand.
Suppose our memory has just three rows, each a length-3 vector, and the
read head's weighting is w = (0.1, 0.7, 0.2). The read
vector r is the average of the rows, weighted by
w — coordinate by coordinate. Watch the second row dominate,
because it carries 70% of the weight:
Three memory rows, and the head's weighting
Each row is a stored vector; the weighting says how much of each to mix.
Mix coordinate 1
Multiply each row's first number by its weight, then add.
Mix coordinates 2 and 3 the same way
One weighted sum per coordinate gives the whole read vector.
The read vector
The result leans hard toward row 2 — exactly because row 2 held 70% of the weight.
The formula you just executed
with Σi wt(i) = 1 and 0 ≤ wt(i) ≤ 1
The six little multiplications you did, written once. That's the entire read operation.
Step 5 · Writing
A write is two soft strokes: first erase, then add.
Reading blends rows out. Writing has to change them — and to stay smooth, it can't just overwrite a row. The paper borrows a trick from the LSTM's forget gate: erase some of what's there, then add something new, both scaled by the weighting.
The write head emits two vectors: an erase vector e
(each entry in 0–1, how much to wipe per coordinate) and an
add vector a (what to deposit). For a single row with
write weight w, the row first fades by w·e, then
gains w·a. Let's do one coordinate of one row, with full
write weight w = 1:
Start: the old value
One coordinate of one memory row currently holds 4.
Erase — fade by w·e
With erase e = 0.5 and weight w = 1, the value is scaled by (1 − w·e) = 0.5.
Add — deposit w·a
With add a = 3, the head writes w·a = 3 on top of the faded value.
New value
Erase-then-add turned 4 into 5 — and where w is small, the row is barely touched, so writes stay local and smooth.
The two formulas you just executed
Erase first, then add. Both multiply by the same weighting w, so a row with tiny weight is left almost untouched.
Step 6 · Content addressing
Way one to point: describe what you want, and rows that match light up.
Reading and writing both need a weighting. Where does it come from? The first of two answers is content-based addressing — point by describing the value you're after.
The controller emits a key k — a query vector. The
head compares k to every row using cosine
similarity (the angle between two vectors: +1 if they point the same
way, 0 if perpendicular). Then a key strength β ≥ 0
scales those scores, and a softmax (turn scores into
positive numbers that sum to 1) produces the content weighting.
The knob that matters is β. Small β makes the
weighting flat — the head hedges across many rows. Large β
sharpens it onto the single best-matching row, approaching a hard lookup.
Drag it and watch:
Drag β. The same four cosine scores; only the focus changes.
What to watch: at β = 0 every row gets 0.25 — the head can't tell them apart. Crank β up and the weight piles onto row 7, the best match. That is content addressing: find rows that resemble a query.
JavaScript is off — the bars show a fixed β = 3 example.
Content weighting, in one line
cosine similarity K[u, v] = (u · v) / (‖u‖ ‖v‖)
A softmax over cosine scores, temperature-scaled by β. Exactly the slider you just dragged.
Step 7 · Location addressing
Way two to point: ignore content, and just step to the next row.
Content addressing finds rows by value. But copying needs something content can't give: after reading row 3, go to row 4 — regardless of what row 4 contains. That is location-based addressing, and it arrives as three small stages bolted after the content weighting.
Together they form the full addressing pipeline. Step through it for a read head walking along the tape during a copy. Each press advances one stage and shows what it does to the weighting:
Press Next to run the four stages on a head copying row by row.
Stage 1 · content — the weighting starts focused on row 3, where the head is now.
What to watch: the gate keeps the previous focus, the shift slides the whole bump one row down to row 4, and sharpening cleans up the blur so the head lands cleanly on the next slot. That is how an NTM walks a list.
JavaScript is off — the bars show stage 1 (content) only.
gate g
Blend new content weighting with last step's weighting. g = 1 uses content; g = 0 keeps the old focus.
shift s
Circularly convolve with a shift distribution — move the bump to a neighbour (−1, 0, +1).
sharpen γ
Raise to power γ ≥ 1 and renormalize — undo the blur the shift introduced.
Step 8 · The pipeline, compressed
Four stages, four short equations — and you've already run all of them.
You dragged β to make content focus; you stepped the gate, shift and sharpen to walk the tape. Here is that exact sequence written down. None of it is new — it's the labels for moves you already made.
k, β, g, s, γ are all emitted by the controller — so the whole addressing scheme is learned, not fixed.
A single head can do pure content lookup (set g = 1, no shift), pure iteration (set g = 0, shift = +1, to march down the tape from wherever it was), or any blend. Copying uses content to find the start, then location to step through — and the controller learns when to use which.
Did it actually learn to compute?
The evidence, and what it set in motion.
Step 9 · The report card
It generalizes by length — the signature of a learned algorithm.
We've assembled the whole machine. The decisive test isn't accuracy on the training set — it's what happens on inputs longer than anything it ever saw. A lookup table can't extrapolate; a learned procedure can.
On the copy task the NTM was trained only on sequences up to length 20. Asked for length 50, 100, even 120, it keeps copying with only mild degradation — because it learned “write each item, then walk back and read.” A 3-layer LSTM baseline, with far more parameters, falls apart the moment the sequence runs past its training range.
Copy task · length generalization (train ≤ 20)
bars are illustrative of the paper's qualitative result; the NTM extrapolates, the LSTM does not.
Copy isn't the only trick. The same machine, with no architectural change, learned a small zoo of algorithms purely from input-output examples:
| Task | What it must learn | Result |
|---|---|---|
| Copy | store a sequence, replay it in order | generalizes far past training length |
| Repeat copy | copy a sequence a given number of times | learns nested loop; counting is the weak spot |
| Associative recall | given an item, return the next one | content lookup, robust past training length |
| Dynamic n-grams | predict next bit from recent history | approaches the Bayesian-optimal predictor |
| Priority sort | sort vectors by an attached priority | learns a priority→location map, then reads in order |
Controllers vary by task: copy uses a feedforward controller (1 head, 100 hidden units); associative recall uses an LSTM controller; priority sort uses 8 parallel heads and 512 hidden units. The memory matrix stays 128 × 20 throughout.
Step 10 · Can you re-explain it?
If you can answer these four, you can teach the NTM.
That's the whole machine: a controller, a memory matrix, soft read/write heads, and a two-way addressing scheme. Close the book and check yourself — answers fold open.
Why can't a normal computer's memory access be learned by backprop?
Reading “address 7” is a discrete choice — a tiny change in the controller either flips the chosen row or doesn't. There's no slope to follow, so gradients can't flow. The NTM replaces the hard pick with a soft weighting over all rows, which moves smoothly.
What is a “weighting”, and how is it used to read?
One non-negative number per memory row, summing to 1. To read, you take the weighted average of the rows: r = Σᵢ w(i) M(i). To write, the same weighting scales an erase-then-add edit across the rows.
What's the difference between content and location addressing?
Content addressing points by value: a key is compared to every row by cosine similarity, scaled by β, softmaxed. Location addressing points by position: a gate, a circular shift, and sharpening let the head step to a neighbouring row regardless of its contents — which is what makes iterating over a list learnable.
Why is length generalization the headline result?
Because it shows the network learned a procedure, not a lookup table. Trained on copy sequences up to length 20, the NTM still copies length 120; an LSTM with far more parameters cannot extrapolate past its training range.
Take the β slider from Step 6 and the addressing stepper from Step 7. Trace, for one full copy of a 3-vector sequence, when the head would set g = 1 (find the start by content) versus g = 0 with shift +1 (march down the tape). That hand-off between content and location is the entire copy algorithm the NTM discovered on its own.
“Our architecture is differentiable end-to-end and can be trained with gradient descent … it learns simple algorithms from example data.” — Graves, Wayne & Danihelka, 2014
The first differentiable computer — and a preview of attention.
The NTM was delicate to train and rarely deployed as-is, but the idea it proved — soft, differentiable access to a stored set of vectors — became one of the load-bearing pillars of modern AI.
Differentiable Neural Computer
The same authors scaled the idea up with richer memory access, solving graph and reasoning tasks — published in Nature.
Transformer attention
Kept the soft-addressing insight — a weighting that sums to 1 over a set of vectors — and dropped the explicit memory tape, attending over the tokens themselves.
Memory & retrieval
Memory networks, retrieval-augmented generation, and external-memory agents all trace to the same root: computation and storage, decoupled and differentiable.
Before attention was “all you need,” the Neural Turing Machine showed that a network could learn where to look — and learn an algorithm in the process.