Most memory networks treat their slots like a filing cabinet: each one holds a fact, and recall means pulling the right drawer. This paper asks a sharper question — can those stored memories reason about each other? The answer it lands on is to take the same self-attention that powered the Transformer and run it inside the recurrent state, letting memory interact with memory at every timestep. That single move reframes memory from storage into a small relational workspace.
Key Findings
- Memories that talk to each other. The Relational Memory Core (RMC) applies multi-head dot-product attention across its own memory slots, so the network can compose relationships between remembered entities rather than read them out independently — exactly where plain LSTMs stumble.
- Diagnosed before it fixed. The authors first show standard memory architectures genuinely fail on relational tasks (like sorting by distance), making the case that this is a real deficit, not a contrived benchmark.
- Broad, not narrow, gains. The same module improves RL (Mini Pacman), program evaluation, and language modeling — setting new records on WikiText-103, Project Gutenberg, and GigaWord, which is unusual reach for one architectural tweak.
How It Works
Instead of attending over an external input sequence the way a Transformer does, the RMC turns attention inward: the fixed set of memory vectors attends to itself and to new inputs, updating through a gated recurrent step. It is the bridge between recurrence and attention — recurrence keeps the compressed temporal state, attention supplies the relational compute.
Great Fit / When to Skip
Great fit if you want to understand how attention migrated from the Transformer into recurrent models, or you care about relational reasoning over sequences specifically. Look elsewhere if you are after a modern long-context recipe — by 2018's standards this was strong, but pure attention-based and state-space models have since absorbed most of these gains without needing a recurrent core.
