Don't hope the network compares objects. Wire the comparison in.
By 2017 convolutional networks could name almost anything in a picture. But ask one a relational question — "is the cube the same material as the cylinder?" — and a strong convnet simply can't answer it reliably, no matter how long you train it. The capacity is there; what's missing is structure. Nothing in a plain network forces it to actually put two objects side by side and compare them.
A Relation Network (RN) bolts that structure on. It takes a set of
objects, runs the same tiny function gθ
over every pair of them, sums all the pair results into one
vector, and passes that to a second small network
fφ for the answer. The whole module is one line:
RN(O) = fφ(Σ gθ(oi, oj)).
That hard-wired "look at all pairs" is a deliberate
inductive bias — a built-in assumption about
how to solve the task — not extra depth or parameters.
The constraint is the feature: forcing every pair through one shared function is what makes relational reasoning happen at all.
The gap
A strong convnet that still can't compare two objects.
Step 1 · The black box & the task
A scene plus a question goes in; an answer comes out.
Here is the whole task in one picture. The input is two things: a small scene of colored shapes, and a question about it. The box must output a one-word answer. This exact example — six objects and the question "What shape is the object closest to the gray one?" — will follow us through all eight steps.
This kind of task is called visual question answering (VQA) — answer a natural-language question about an image. The paper's headline result is on CLEVR, a VQA benchmark built precisely to require reasoning. With a Relation Network attached, the model reached 95.5% — above the 92.6% humans scored, and far above the 68.5% of the previous best model on the same pixel inputs.
From the abstract: “…we achieve state-of-the-art, super-human performance” on CLEVR. CLEVR images contain 3D-rendered shapes; Sort-of-CLEVR (our simplified running scene) uses flat 2D squares and circles.
Step 2 · Two kinds of question
Some questions read one object; some compare several.
You've seen the box answer a question about our scene. Before opening it, notice that not all questions are the same kind — and the difference is the whole point of the paper.
A non-relational question can be answered by looking at one object: "what color is the square at the top?" You find that object and read off an attribute. A relational question forces you to compare objects: "what shape is the object closest to the gray one?" You can't answer it from any single object — you must measure the gray object against each of the others and pick a winner.
Non-relational · read one object
One object holds the answer. A plain convnet handles these fine — about 94% right.
Relational · compare across objects
No single object answers it — you must compare distances. This is what a plain convnet can't do.
Same scene, same network. The non-relational question is easy; the relational one needs a comparison between objects. The paper's whole argument is that this second ability doesn't come for free with more capacity — it needs to be built into the architecture. Step 3 shows what happens when it isn't.
Sort-of-CLEVR (the paper's diagnostic dataset) puts 6 objects of 6 colors in each scene and asks 10 relational and 10 non-relational questions per image — deliberately separating the two skills.
Step 3 · The wall a plain convnet hits
It aces the easy questions and stalls on the hard ones.
Step 2 split the questions into two kinds. Now run a strong, ordinary model on both — and watch it split right down that same line.
The model here is a convnet that reads the scene, plus an MLP (multilayer perceptron — a plain stack of fully-connected layers) that reads the convnet's features and the question, then outputs the answer. No relational structure, just raw capacity. On Sort-of-CLEVR the authors measured each question type separately:
Non-relational questions
Both ≈ 94%. The plain model has no trouble reading a single object.
Relational questions
CNN+MLP stalls at 63% — barely above guessing. Add the RN and it jumps to 94%.
Read the two charts together and the gap is unmistakable. The same convnet features, the same training, the same scenes — yet the plain model is fine on one column and stuck on the other. It isn't a data problem and it isn't a capacity problem: on the hardest relations, like "closest-to", the plain model scored only about 52%, no matter how much it trained.
This is the experiment that makes the paper's claim precise. It isolates relational reasoning as a distinct, measurable skill — one that a high-capacity convnet simply lacks on its own.
A plain network finds it hard to compare objects in pairs — even when it has the capacity. If we could make "consider every pair" the built-in default instead of something the model must discover, the gap would close. That is exactly what Part B's module does.
Sort-of-CLEVR results, Section 5.3. Only the model's back end differs between the two bars (MLP vs RN); the convnet that reads the image is the same — a clean apples-to-apples comparison.
The module
What an object is · compare every pair · the one-line formula
Step 4 · What is an “object”?
The grid cells of a feature map become the objects.
Step 3 said the network needs to compare objects. But the scene is just pixels — where do the "objects" come from? The RN never detects or labels them. It takes whatever set the encoder hands it.
A convnet turns the image into a feature map: a small grid of
cells, each a vector summarizing what's at that spot. The RN simply
declares every cell to be an object. The CLEVR model uses four
conv layers that produce an 8×8 grid with
24 channels — so 64 cells, each a
24-number vector. A cell may cover one shape, part of one, or empty
background; the module doesn't care.
A bare feature vector says what is at a cell but not where. Appending the cell's (x, y) position lets the pair function reason about spatial relations like "left of", "closest to" or "furthest from" — which our running question needs.
The same trick generalizes: for the bAbI text task the "objects" are the LSTM states of each supporting sentence; for the physics task they are the per-ball state vectors. Whatever the encoder emits, the RN treats it as a set of objects.
Step 5 · Compare every pair, then sum
One shared function scores each pair; add them all up.
Step 4 gave us a set of objects, each tagged with where it sits. Here is the engine of the whole paper — and before any formula, you should run it by hand.
Pick the four objects nearest the action in our scene: the
gray anchor and three others. The RN forms every
unordered pair of objects and feeds each pair
(plus the question) through the same little network
gθ. For our "closest-to-gray" question,
gθ has learned to output a high score when a
pair is "the gray one next to a near neighbour". Below are the toy
scores it produces — bigger means "more relevant to the answer":
Toggle which pairs the RN sees — watch the sum and the answer change
4 objects · 6 pairs
Sum of 6 active pairs: 1.80 → fφ reads it and answers square (red is closest)
Turn off the “gray ↔ red” pair and the strongest piece of evidence vanishes — the sum drops and the answer is no longer reliable. That is the point: the RN doesn't single out the right pair in advance. It scores all of them and lets the sum carry the signal, so the relevant comparison is always somewhere in the total.
The same weights score every pair, so the network learns
"how to compare two objects" once and reuses it everywhere.
And because the pair scores are summed, the result doesn't
depend on the order of the objects — a set, not a list. With
n objects there are n(n−1)/2 pairs: 4 objects
give 6, the full 64-object CLEVR grid gives 2,016.
The scores here are a toy stand-in for what gθ actually outputs — a 256-d vector per pair, not one number. The real sum is over all ordered pairs i, j; we use unordered pairs to keep the hand count small. The mechanism — score every pair, sum — is exactly this.
Step 6 · The one-line formula
Everything you just did, folded into one line.
In Step 5 you scored every pair and summed them, then let a second network read the sum and answer. The formula is simply a receipt for those three moves — no new magic.
RN(O)=fφ(Σi,jgθ(oi,oj,q))
O = your set of objects · gθ = the shared pair function · Σ over all pairs = your six rows · fφ = reads the sum, gives the answer · q = the question vector
Read it right to left and it is exactly the interactive you just used.
gθ(oi, oj, q) is one row of
the pair list — the same little network applied to every pair, with the
question q tacked on so it knows which relation to
look for. Σi,j is the running total at the bottom
of the panel. fφ is the final network that turns
that total into the answer "square".
Both gθ and fφ are plain
MLPs. In the CLEVR model, gθ is a 4-layer MLP of
256 units each; fφ is a 3-layer MLP
(256, 256, then the answer logits). Tiny — and trained end-to-end with
the convnet by ordinary backpropagation.
The summation makes the output order-invariant — shuffle the objects and the answer is unchanged, exactly what a set of objects demands. And because gθ always sees objects two at a time, the network cannot skip the comparison. That is the inductive bias, written in one line.
Equation 1 of the paper, with the question conditioning folded in: gθ's input is the concatenation [oi, oj, q]. The sum runs over all ordered pairs; fφ ends in a softmax over the answer vocabulary.
Step 7 · The whole pipeline
A module that bolts onto the back of any encoder.
You now have every piece: objects (Step 4), the all-pairs sum (Step 5), the formula (Step 6). Snap them together and you can see why the authors call it a plug-in module, not a new model.
Front to back: a convnet turns the image into objects, an LSTM turns the question into a vector, the RN compares all object pairs and sums, and a final MLP reads the sum and answers. Only the middle box is new — the encoders are off-the-shelf and the whole thing trains end-to-end.
There is one real cost. With n objects the RN runs
gθ on n² pairs, so compute grows
quadratically. For CLEVR's 64-cell grid that is 4,096 pair
evaluations per question — fine here, but a real ceiling as scenes grow.
That quadratic cost is exactly what later attention-based models would
learn to manage.
No new optimizer, no task-specific tricks, two small MLPs. The power comes entirely from the structure — forcing all-pairs comparison — not from scale. That is the paper's whole thesis in one diagram.
CLEVR-from-pixels model: 4 conv layers × 24 kernels (3×3, stride 2, BN, ReLU); a 128-unit LSTM with 32-d word embeddings for the question; gθ = 4×256 MLP; fφ = 256, 256 (50% dropout), then the answer logits.
The payoff
Did it work — and can you now explain it?
Step 8 · Results & recap
Did it work? — and can you now explain it?
You've built the whole idea: two kinds of question (Step 2), the gap a plain net hits (Step 3), objects (Step 4), all-pairs sum (Step 5), the formula (Step 6), the pipeline (Step 7). Final question — how far did it actually go?
Report card · CLEVR accuracy % · higher is better
CNN+LSTM+RN · this paper
Human baseline
CNN+LSTM+SA · prior best
CNN+LSTM · no relations
0CLEVR accuracy % · higher is better100
One number tells the whole story: on relational questions a plain CNN+MLP sat at 63%; bolt on the RN and the same model jumps to 94%. The module didn't add capacity or data — it added the structure to compare objects, and that was the missing piece all along.
Now you can explain it. Five questions — answer each out loud before opening it. If all five come easily, you've genuinely got this paper.
Why can't a strong convnet answer relational questions?
It has the capacity, but nothing in its architecture forces it to put two objects side by side and compare them. On Sort-of-CLEVR it aced single-object questions (≈94%) yet stalled on relational ones (≈63%), no matter how much it trained — the gap Step 3 isolated.
What does an RN actually compute?
RN(O) = fφ(Σi,j gθ(oi, oj, q)). It runs one shared little network gθ over every object pair, sums all the results into one vector, and lets fφ read that sum and produce the answer — exactly the panel you toggled in Step 5.
What counts as an “object”?
Whatever the encoder emits. For CLEVR, the cells of an 8×8×24 feature map — 64 objects, each tagged with its (x, y) position. The RN never detects or labels objects; it just treats the set of cells (or LSTM states) as objects to compare.
Why sum over all the pairs?
The sum makes the output order-invariant — a set, not a list — and guarantees the relevant comparison is included, since every pair is scored. The cost is quadratic: n objects mean n² pair evaluations, 4,096 for CLEVR's 64-cell grid.
Why is it called a “simple module”?
It is two small MLPs (gθ and fφ) that bolt onto the back of an off-the-shelf CNN or LSTM, with no new optimizer and no task-specific tweaks. The same module worked across vision, text and physics — the power is the all-pairs structure, not scale.
What happened next — "compare all pairs, then aggregate" became one of the load-bearing ideas in deep learning. It is the same shape as self-attention (which compares all pairs with learned weights instead of a fixed sum) and graph networks, and it anchored DeepMind's "relational inductive biases" line of work. The lesson outlived the module.
“Our results show that relation networks are a simple, plug-and-play module that can solve problems requiring rich relational reasoning.” — the abstract, Santoro et al., 2017
“Compare all pairs, then aggregate” is everywhere now.
Relation Networks didn't stay the dominant tool for long — but the idea they crystallized did. Forcing a model to reason over a set of objects, pair by pair, became one of the organizing principles of modern deep learning, and it shows up — generalized — in the architectures that replaced the RN itself.
Self-attention
Attention is the RN's all-pairs comparison with a twist: instead of a fixed sum over pairs, it learns soft weights and skips the explicit gθ. The same "every token looks at every other" structure powers every Transformer.
Graph networks
Message-passing graph neural networks generalize the RN: objects become nodes, pairs become edges, and the shared pair function becomes an edge update. DeepMind's "relational inductive biases" paper made the lineage explicit.
Structure over scale
The headline isn't 95.5% — it's that the right architectural prior beats raw capacity on a whole class of problems. "Wire in the computation you need" became a default move, not a clever exception.
One shared function over every pair — and a skill a plain convnet couldn't learn became something you could simply build in.