Don't just store memories — let them reason about each other.
A recurrent network remembers the past by squeezing it into a hidden state. Slot-based memories do a little better: they keep a few separate drawers, each holding a fact. But in both cases the stored pieces just sit there. Asking how two remembered things relate — which is farther, which came first, which depends on which — is exactly where they stumble.
The authors made one bet: take the very same self-attention that powered the Transformer and run it inside the recurrent state. Now the memory slots are not isolated drawers — at every timestep each slot attends over all the others and updates itself from what it finds. Memory stops being storage and becomes a small relational workspace.
Recurrence keeps the compressed timeline; attention, turned inward, supplies the relational compute.
The machine from orbit
What problem is this, and what shape is the answer?
Step 1 · The black box
A box that reads a sequence — then reasons about it.
Here is the whole story in one picture. A stream of vectors arrives one at a time, the box remembers them, and at the end it answers a question about how those vectors relate. This exact toy problem — the paper's own diagnostic, the N-th Farthest task — will follow us through all ten steps:
The task sounds almost trivial, yet it is brutal for a normal memory. The model never sees the distances — it must compute every pairwise distance between the eight vectors, then implicitly sort them, and only then read off the n-th one. As the paper stresses, it must sort distance relations between vectors, not the vectors themselves.
That is the deficit the paper is after. A standard LSTM packs everything into one hidden vector — a great compressor, a poor relational reasoner. On this task it never gets past 30% accuracy. The Relational Memory Core reaches 91%. The rest of this page opens the box to see why.
From the abstract: “standard memory architectures may struggle at tasks that heavily involve an understanding of the ways in which entities are connected — i.e., tasks involving relational reasoning.”
Step 2 · Why the old memory fails
A filing cabinet can store facts — it can't make them talk.
You've seen the task: relate eight remembered vectors. Now look at why the standard ways of remembering fall short — it tells us exactly what the new module has to add.
There are two classic ways to carry the past forward. An LSTM (a recurrent net that learns what to keep) crams everything into one hidden vector. A slot-based memory keeps several separate cells, each holding a fact. Compare three pictures — the third is the gap this paper fills.
Way 1 · one hidden vector
A great compressor — but the eight vectors are now mashed into one blur. Picking two back out to compare is hard.
Way 2 · separate drawers
Now facts are kept apart, which is tidier. But the drawers never exchange anything — slot 1 has no way to ask about slot 3.
The gap · let them talk
The fix is one missing ingredient: a channel that lets each slot read every other slot. That channel is self-attention.
Storage and retrieval were already solved. The unsolved part is interaction — letting two remembered things influence each other. Everything new in this paper is about adding that one missing channel between memory slots.
From the paper: standard architectures “pack all information into a common hidden memory vector, potentially making compartmentalization and relational reasoning more difficult.”
Step 3 · The shape of the machine
The memory is a grid of slots — and it loops back into itself.
We know the missing piece is interaction between slots. Before we build it, let's fix the shape it lives in — what the memory actually looks like, and how it carries across time.
The memory is a matrix M: N rows (the memory
slots) by F columns (numbers per slot). In the paper's
language-modeling setup, for example, the total memory is 2500 units
split across the slots. Each row is one little workspace.
And it is recurrent: at every timestep the new input vector
xt arrives, the core mixes it into the
memory to produce an updated Mt, and that
same matrix is fed back in at the next step. Same shape out as in —
that is what lets the loop run forever.
Two questions are still open: how the core mixes the slots (Part B's core mechanism), and how it decides what to keep versus overwrite (the gate, Step 8). The shape — a looping grid of slots — is the stage all of that plays out on.
Memory talks to memory
Feel it · compute it by hand · compress it into one line · add the input · close the gate
Step 4 · Pass 1 — feel it
Pick a slot, and watch it pull from the slots it cares about.
We have a grid of slots that loops over time. Now for the one move that makes them a workspace instead of a cabinet — and before any math, you should feel what it does.
Suppose the eight vectors of our task are spread across eight memory slots, plus a slot holding the question. The model wants the 2nd farthest from v7. The slot for the answer needs to know how far every vector sits from v7 — so it must read the other slots and pull in what's relevant.
That is self-attention turned inward: each slot scores every other slot for relevance, then blends the useful ones into itself. Pick a slot below and watch which slots it leans on:
Hover / tap an underlined slot
Arc thickness = attention weight. The answer slot reads the reference v7 and the candidate vectors, comparing their distances; the question slot tells everyone which vector is the reference. This mirrors the paper's analysis: once the reference is seen, slots preferentially attend to wherever it was written.
Second exhibit · before the reference is seen
From the paper's attention analysis: before the reference vector arrives, the model shuttles incoming vectors into one or two staging slots — high attention from those slots' queries to the input — holding them until the question lands.
Arc weights here are illustrative; the qualitative patterns follow the paper's analysis of the RMC's attention on the N-th Farthest task (its Figure 3).
Step 5 · Pass 2 — compute it by hand
Six small steps. Real numbers. No magic left.
You've felt one slot pull from others. Now do it yourself, with actual numbers, so there's nothing left to take on faith.
We shrink the memory to two slots — slot A holds vector v5, slot B holds vector v3 — and follow the answer slot through one round of attention as it looks for “far from v7”. Six rows; each one is a single arithmetic move.
Make a query, a key, and a value for each slot
Multiply each slot's row of numbers by three learned matrices — Wq, Wk, Wv — to get three vectors. The answer slot's query asks “who is far from v7?”; each slot's key advertises “here's my distance to v7”; its value is “the identity I hand over if picked.”
Score: dot the query against every key
A dot product multiplies two vectors position by position and adds it all up — a big result means the directions align. It's our relevance meter: how well does each slot's key answer the query's question? v5 sits far from v7, so its key aligns more.
Scale: divide by √dk
Here dk = 64, so √dk = 8. With many positions summed, dot products grow large — and huge scores park the next step (softmax) in a flat region where learning gradients die. Dividing by √dk keeps the numbers in the healthy zone.
Softmax: turn scores into weights
Softmax squashes any list of scores into positive weights that sum to 1 — by exponentiating each score and dividing by the total. Note what it did to our gap: a 12-vs-10 score lead became an 88-vs-12 weight lead.
Weigh each slot's value by its weight
Multiply every slot's value vector by its weight. The losing slot (v3) isn't erased — it's shrunk toward zero, still faintly present in the blend.
Sum — and the answer slot is updated
Add the weighted values. The result is the new content of the answer slot: mostly v5 (the far one), with a measured trace of v3 — exactly the relational read-out the task needs.
Drag the scores — watch row 4 happen live
Softmax exaggerates gaps: a small score lead becomes a big weight lead. Push the scores equal and the slot blends both 50/50; open a gap of 4+ and the farther vector takes nearly everything.
A slot mostly keeps itself, but measurably blends in what it needs from the others. That blend — recomputed fresh at every timestep — is how a memory reasons about its own contents.
Checking row 4: e¹²/(e¹² + e¹⁰) = 1/(1 + e⁻²) ≈ 0.88. Every slot runs this same pass with its own query — so every slot gets its own update.
Step 6 · Pass 3 — compress it
Your six steps, folded into the paper's one line.
You just did six steps for one slot. Stack every slot's row into the memory matrix M and project the whole thing at once — and all six steps for all slots collapse into the paper's equation (1):
Aθ(M) = softmax(MWq(MWk)ᵀ√dk)MWv
MWq=Q, MWk=K, MWv=V · QKᵀ = rows 2–3 · ÷√dk = row 3 · softmax = row 4 · ·V = rows 5–6
This formula is not new information — it's a receipt for work you
already did by hand. The only addition is bookkeeping:
MWq(MWk)ᵀ dots every slot's
query against every slot's key at once, producing the
all-pairs score table. Notice it's the same matrix M feeding
all three projections — that's what makes it self-attention.
See the same six steps in matrix form
row 1 → Q = MWq, K = MWk, V = MWv
rows 2–3 → S = QKᵀ / √dk (every slot-pair score, scaled)
row 4 → A = softmax(S) (each row sums to 1)
rows 5–6 → M̃ = AV (each row of M̃ is one slot's update)
The key contrast with a Transformer: there the rows of M are words in the input; here they are memory slots in the recurrent state. Same equation, attention turned inward.
Step 7 · Multiple heads
Run several attentions at once — each a different kind of relation.
You now know how one round of attention mixes the slots. The RMC runs several in parallel — the paper calls them heads — here's why.
“Far from v7” is one relation, but a slot might need several at once:
who is far, who is near, who came first. A single head has one weight
budget that sums to 1, so it must average those into a muddier blend.
The fix: run h independent attention units in parallel,
each with its own learned Wq, Wk,
Wv. Each head specializes; their outputs are then
concatenated back into one slot. The language model used
h = 4 heads.
Pick a head, watch the slot-to-slot pattern switch
Holds its own content
Each slot attends mostly to itself — keeping its stored vector intact.
Patterns illustrative. The point is that different heads can carry different relations between the same slots — one tracks “who is far”, another “what arrived recently” — and they don't have to compete for one weight budget.
From the paper: “heads could be useful for letting a memory share different information, to different targets, using each head.” The language-modeling RMC used 4 heads and 1 attention block.
Step 8 · The gated recurrent update
Read the new input, then decide how much to keep — like an LSTM.
We can now mix slots with each other. Two things are still missing: how a brand-new input gets in, and how the memory decides what to keep versus overwrite. This step adds both.
Letting the input in. At each timestep the new vector
xt is stacked under the memory — the paper
writes this [M; x] — and the slots attend over that
combined set. Because only M provides the queries, the output
M̃ keeps M's shape: the same self-attention now both
relates slots and pulls in the new observation.
Deciding what to keep. The proposed update M̃ doesn't simply replace the old memory. It is fed through the same gates an LSTM uses — a forget gate that scales the old memory and an input gate that scales the new update — so each slot blends old and new under learned control:
mt = σ(ft)∘mt−1 + σ(it)∘gψ(M̃t)
σ = sigmoid (a 0→1 dial) · ∘ = element-wise · σ(f) keeps the old · σ(i) admits the new · gψ = a small per-slot MLP with LayerNorm
A grid of slots, updated each timestep by self-attention over
[M; x], then committed through LSTM gates. That is the
Relational Memory Core: recurrence keeps the compressed timeline;
attention, turned inward, supplies the relational compute.
From the paper: “we found a row/memory-wise MLP with layer normalisation to work best” for gψ; output gates were not found necessary.
Does it work, and what did it change?
The scorecard, a quick self-test, and the lasting impact
Step 9 · The scorecard
One module, the same gain across three very different jobs.
You've built the whole RMC. Now the question that matters: does letting memories interact actually help? The paper tests it where the deficit was clearest, then on real workloads.
Start with our running example. On the N-th Farthest task — pure relational reasoning over the eight vectors — a standard LSTM and even the DNC (a sophisticated external-memory net) stall below 30% accuracy. The RMC reaches 91%. That gap is the headline: the missing channel between slots was the whole story.
Best batch accuracy · 16-dim vectors
0%100%
The surprise is the reach. The very same module set new best results on three language-modeling corpora at once — Wikipedia, books, and news — lowering perplexity (the average “how surprised was the model” score; lower is better) by 1.4–5.4 points, a 5–12% relative improvement. One architectural tweak rarely helps that broadly.
| Model | WikiText-103 | Gutenberg | GigaWord v5 |
|---|---|---|---|
| Best prior LSTM | 34.3 | 45.5 | 43.7 |
| Relational Memory Core | 31.6 | 42.0 | 38.3 |
The attention analysis on N-th Farthest matched intuition: before the reference vector appears, slots stage incoming vectors; once it appears, slots preferentially attend to wherever it was written.
Step 10 · Now you teach it
If you can answer these, you can re-explain the RMC.
You've walked the whole machine — the shape, the slot attention, the gate, the results. Here's the test: cover the answers and say each one out loud first.
In one sentence, what is the Relational Memory Core?
A recurrent memory of N slots that, at every timestep, lets each slot attend over all the others (and the new input) with multi-head self-attention, then commits the result through LSTM gates. Storage plus interaction.
Why does a plain LSTM fail at N-th Farthest?
It packs the whole sequence into one hidden vector, blending the eight vectors together. To answer it must compare pairs of remembered vectors — but they're no longer separable, so it can't compute the pairwise distances and sort them.
What does the self-attention compare — slots, or words?
Slots. The exact same equation as the Transformer — softmax(QKᵀ / √dk)V — but the rows of M are memory slots in the recurrent state, not words in an input sentence. Attention turned inward.
How does a new input get into the memory?
It's stacked under the memory as [M; x] and attended over. Only M supplies the queries, so the output keeps M's shape — the same attention both relates slots and reads the new observation.
What do the forget and input gates do?
They split the update per slot: σ(f) scales how much old memory to keep, σ(i) scales how much of the new attention-based update to admit. mt = σ(f)∘mt−1 + σ(i)∘gψ(M̃). Recurrence holds the timeline; attention supplies the relations.
Why is the broad result more telling than any single one?
The same module improved RL (Mini Pacman), program evaluation, and three language-modeling corpora at once. One architectural change rarely helps that widely — it suggests relational interaction was a real, general deficit, not a benchmark quirk.
What happened next — the RMC showed that the Transformer's self-attention could live inside a recurrent state. The field largely went the other way — dropping recurrence for pure attention and, later, state-space models — but the core insight, that memory should reason about its own contents, runs straight through to today's long-context models.
“We then improve upon these deficits by using a new memory module — a Relational Memory Core (RMC) — which employs multi-head dot product attention to allow memories to interact.” — the abstract, Santoro et al., 2018
A bridge between two eras of sequence models.
This paper sits at the hinge between recurrence and attention. It proved self-attention could live inside a recurrent state — and made a lasting case that memory should reason about its own contents, not just store them.
Attention as relational compute
It framed multi-head self-attention as a general engine for relating entities — a view that powers relational reasoning across graphs, sets and agents.
Pure attention won out
Within a couple of years, models mostly dropped the recurrent core entirely — Transformer-XL, then large language models — absorbing these gains without a loop.
Memory that thinks
State-space and modern long-context models still chase the same goal: a compact state that can reason over what it holds. The RMC named the problem early.
Recurrence keeps the timeline; attention, turned inward, supplies the relational compute — an idea that outlived the architecture that introduced it.