Don't design the features. Let the network discover its own — by sending the error backward.
A network with no middle layer can only map similar inputs to similar outputs. To do anything harder it needs hidden units — neurons in between that build their own re-coding of the input. The catch, and the reason neural networks had stalled for a decade: nobody knew how to train those hidden units. They sit in the dark, with no “correct answer” of their own to aim at.
This paper's bet is one mechanism, called error propagation (today: backpropagation). After the network guesses, you measure how wrong it was, then push that error backward through the very same wires it travelled forward on. Every weight — even one buried deep inside — learns how to change using only the signals available right next to it. The lasting idea isn't the math of gradient descent, which is older; it's that a machine can discover its own internal representations instead of being handed features by people.
“The major theoretical contribution … is the procedure called error propagation, whereby the gradient can be determined by individual units of the network based only on locally available information.” — from the abstract
The problem worth a decade
What hidden units are for · why XOR breaks a simple network · the gap nobody could close.
Step 1 · The black box that learns
A box you teach by example — show it inputs and the answers you want, and it programs itself.
Here is the whole machine in one picture. Numbers go in at the bottom, flow up through a layer of hidden units — neurons that are neither input nor output — and a prediction comes out the top. You never write the rule by hand. You show it example pairs of (input, desired output), and it adjusts its own connection strengths until it reproduces them.
To make the rest of the page concrete, we'll teach this box one specific, famous little function — the one that exposed exactly what a network without hidden units cannot do.
Our running example is XOR (“exclusive-or”): two binary inputs,
output 1 when they differ and 0 when
they match. Trivial to state. As we'll see next, it was the
rock the previous generation of networks broke on.
| Input A | Input B | XOR output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
Step 2 · Why XOR breaks a simple network
No single straight line separates XOR — so you need a hidden unit to re-code the input first.
You have the box and the function. Now meet the obstacle that makes hidden units necessary rather than optional.
A network with no hidden layer draws a single straight boundary: it can only answer questions of the form “is the weighted sum of the inputs above some threshold?” Plot XOR's four cases and the trouble is visible — the two 1s sit on one diagonal, the two 0s on the other. No straight line puts the 1s on one side and the 0s on the other.
The fix: add one hidden unit that detects a new feature — “are both inputs on?” — and let the output subtract it. The hidden unit changes the input's similarity structure enough that a straight line can finish the job. Here is a hand-built network (the paper's Figure 2) that does exactly this. Each unit fires only when its incoming sum clears the threshold written inside it. Try all four inputs and watch it compute XOR.
Flip the two inputs and watch the hidden unit and the output
The hidden unit is a “both-on” detector. The output adds the two inputs (+1 each) and subtracts twice the hidden unit (−2). When both inputs are on, the hidden unit switches on and cancels them — output 0. When just one is on, the hidden unit stays off and the output fires — 1. That is XOR, built from a re-coding no straight line could reach.
Step 3 · The gap nobody could close
The output unit has a target to aim at. The hidden unit doesn't — so how should it ever change?
We just hand-wired those weights. The whole point of a learning machine is that nobody should have to. So: can the network find them on its own?
For the output unit, learning is easy: you know the answer you
wanted. If it should have said 1 and said
0.6, the gap (target − output)
tells you which way to nudge each of its weights. This is the old
perceptron / delta rule, and it works
whenever there's a target.
But a hidden unit has no target. Nobody ever says what it should have output — only the final answer is graded. So when the network is wrong, how much of the blame belongs to each hidden weight? This is the credit-assignment problem, and it was the wall.
In Perceptrons (1969), Minsky & Papert had shown there was no learning rule for the hidden units of a multi-layer machine, and doubted one would be found. That pessimism froze neural-network research for years. Part B is the thaw.
Backpropagation, built up by hand
Learning as rolling downhill · a smooth neuron · the backward pass · one rule, two passes.
Step 4 · Turn learning into rolling downhill
Score how wrong the network is, then change every weight in the direction that lowers that score.
The credit-assignment wall falls the moment we stop asking “what should the hidden unit output?” and ask instead “which way does each weight move the overall error?”
Give the network one number that says how wrong it is. The paper uses the sum-squared error E: take the gap between each target t and actual output o, square it (so over- and under-shoots both count as positive), and add them up.
half the squared gap between target and output, summed over the output units
Now picture every weight in the network as one knob, and E as a landscape sitting above all those knobs — high where the network is wrong, low where it's right. Learning is just rolling downhill: for each weight, find the slope of E in that weight's direction, and take a small step the opposite way. The step size η (“eta”) is the learning rate.
nudge each weight downhill — opposite to the slope of the error
no hidden units
with hidden units
Step 5 · The neuron needs a smooth dial, not a switch
To follow a slope you need a slope — so swap the hard on/off threshold for a smooth S-curve.
Rolling downhill needs the error to have a slope with respect to every weight. That can't happen if a neuron flips instantly from off to on — a cliff has no usable slope.
So replace the hard threshold of Step 2 with the logistic function: a smooth S-curve that eases from 0 up to 1. Its input is the weighted sum plus a bias; its output is squashed into (0, 1).
the logistic squashing function · net = (weighted sum of inputs) + bias
The magic is its slope. The derivative of the logistic has an unusually tidy form — you can read it straight off the output itself:
the slope is biggest when the unit is unsure (o = 0.5) and vanishes when it is confident (o near 0 or 1)
Drag the input — watch the output and its slope
output o = σ(net)
o = 0.50
slope o (1 − o)
slope = 0.25
Learning is fastest for units still on the fence. The slope peaks at 0.25 when the output is exactly 0.5, and fades toward 0 as the unit commits to 0 or 1. Because every weight change is scaled by this slope, a confident unit barely moves — which keeps learning stable.
One consequence: the logistic only reaches 0 or 1 with infinite weights. So in practice the paper aims at targets of 0.1 and 0.9, even while talking as if it wants 0 and 1.
Step 6 · The backward pass — the whole trick
Send the output's error back through the same wires; each hidden unit's blame is its slope times the blame of whatever it feeds.
Now we have everything: a smooth neuron (so slopes exist) and a hill to roll down. The only question left is the hard one from Step 3 — how does a hidden weight learn its share of the blame?
Give every unit an error signal δ (“delta”) — how much the total error would drop if that unit's input were a touch higher. Two cases:
Output unit — it has a target, so its error signal is the gap it made, scaled by how fast its dial was moving: δ = (t − o)·o(1−o).
Hidden unit — it has no target, so it borrows one. Its blame is the blame of every unit it feeds, pulled back through the exact weight it sent forward on, then scaled by its own slope: δ = o(1−o)·Σ δₖ wₖ. That single backward step is error propagation. Apply it from the top down and every layer gets a signal — no targets required.
Pick an input, run the forward pass, then the backward pass
Forward pass — compute the output
Press “Forward pass” to send the input up through the network.
Backward pass — propagate the blame
Then press “Backward pass” to send the error back down.
Watch the input that is 0: its weight change comes out exactly 0, because Δw = η·δ·(input) and a silent input has nothing to learn. And notice the hidden unit — which had no target — still receives a real δ and a real update. That is the wall from Step 3, gone.
Fixed for the demo: w₁ = 0.6, w₂ = −0.4, w₃ = 0.9, hidden bias 0.1, output bias −0.3, learning rate η = 0.5, logistic units. Numbers are rounded for display. This is one untrained step, not a trained net — the point is the flow, not the answer.
Step 7 · One rule, two passes
The whole algorithm: a forward pass to answer, a backward pass to assign blame, one update everywhere.
Step back and the entire method collapses into three short lines and a two-beat rhythm — the same shape every network is still trained with.
the update — identical to the old delta rule: step = rate × error signal × input activity
output unit: blame comes straight from the target
hidden unit: blame is borrowed from the units it feeds, pulled back through its own weights
Forward
Feed the input in; let each layer compute its outputs from the layer below, all the way to a prediction.
Measure error
At the top, compare the prediction with the target to get each output unit's error signal δ.
Backward
Send δ back down, layer by layer, so every unit gets one — then nudge each weight by η·δ·o.
The backward pass costs about the same as the forward pass — it reuses the very same connections. That cheapness is why a network of millions of weights can still be trained today: backprop assigns blame to all of them in a single sweep.
Two practical notes from the paper. Momentum: add a fraction α ≈ 0.9 of the previous weight change to each new one, so the ball keeps rolling through narrow ravines without oscillating. Symmetry-breaking: start from small random weights — if every weight began equal, all hidden units would receive identical blame and stay forever identical.
What it learned, and why it mattered
XOR solved · the feared local minima · hidden units that invent their own features.
Step 8 · It works — and the dreaded local minima rarely bite
Run it on XOR and it finds a solution; run it hundreds of times and it almost never gets stuck.
We built the machinery; does it actually learn? The paper's answer is a long table of simulations. Start with the one we've been carrying.
Turned loose on XOR, the network discovers its own weights — a mirror image of the hand-built one from Step 2, reached after 558 sweeps through the four cases. The real headline is theoretical relief: across hundreds of XOR runs, the descent got trapped in a local minimum only twice. The bumpy surface from Step 4 turned out to be far kinder than feared.
XOR · report card
through the 4 cases, η = 0.5
in a local minimum, of hundreds
The paper is candid about the failures too: the one task where local minima showed up reliably was binary addition, and even there a spare hidden unit or two fixed it. “The problem of local minima has not been serious to date.”
Step 9 · The hidden units invent their own features
Left to learn, the middle layer builds codes nobody designed — and often more elegant than the authors expected.
Solving XOR proves the rule runs. The deeper payoff is what the hidden units become — the “internal representations” of the title.
Symmetry. Asked to tell whether a 6-bit string reads the same forward and backward, the network solved it with just two hidden units. The weights it learned are mirror-symmetric and opposite in sign, in the ratio 1 : 2 : 4 — so a symmetric string sends exactly 0 into both hidden units (they switch off, the output fires), while any non-symmetric string switches at least one on. Nobody designed that code; gradient descent found it.
symmetry detector · learned weights
A symmetric string makes the +/− pairs cancel to net 0.
T-vs-C detector · a shared receptive field
Identical weights tile the image — a learned feature works anywhere.
With more hidden units than strictly needed, the codes stop being tidy “one unit = one concept” detectors. Meaning lives in the pattern of activity across many units, not in any single one — the distributed representation that defines how deep networks store knowledge to this day.
Step 10 · Can you explain it now?
If you can answer these without scrolling up, you can teach backprop to someone else.
One running example — XOR — carried us from “simple networks can't do this” to “here's how every modern network learns.” Test yourself.
Why can't a single-layer network learn XOR?
It can only draw one straight boundary, but XOR's two “1” cases sit on the opposite diagonal from its two “0” cases — no straight line separates them. You need a hidden unit to re-code the input into a form that is separable.
What is the credit-assignment problem?
An output unit has a target, so you know how to correct it. A hidden unit has none — only the final answer is graded. Figuring out each hidden weight's share of the blame for the final error is the problem backprop solves.
What does the backward pass actually compute?
An error signal δ for every unit. For an output unit, δ = (t − o)·o(1−o). For a hidden unit, δ = o(1−o)·Σδw — the blame of the units it feeds, pulled back through its own weights and scaled by its slope. Then every weight changes by η·δ·o.
Why the logistic instead of a hard threshold?
Rolling downhill needs a slope, and a step function has none — it's flat then a cliff. The logistic is smooth, and its slope has the tidy form o(1−o): biggest when the unit is unsure, near zero when it's confident.
What are the two passes, in one breath?
Forward: feed the input up to a prediction. Backward: compare with the target, then send the error back down so every unit gets a δ. The backward pass costs about as much as the forward one.
What did the symmetry and T–C experiments show?
That hidden units learn useful features nobody programmed — a mirror-symmetric 1:2:4 code for palindromes, shared receptive fields for position-independent shape detection. Features can be discovered, not hand-designed.
What happened next — this learning rule, scaled up on fast hardware and large datasets, became the training engine of essentially every neural network since: convolutional nets, recurrent nets, Transformers, and the large language models of today. The paper you just read is where it starts.
“We believe that we have answered Minsky and Papert's challenge and have found a learning result sufficiently powerful to demonstrate that their pessimism about learning in multilayer machines was misplaced.” — the conclusion, Rumelhart, Hinton & Williams
Every network trained today still learns the way this page just showed.
One idea — send the error backward — reopened a research program that had been declared a dead end, and grew into the default way machines learn from data.
Answered Minsky & Papert
It supplied the missing learning rule for hidden units and reignited neural-network research after the long “perceptron winter.”
How deep learning trains
Backprop plus gradient descent is the core of training CNNs, RNNs, Transformers and LLMs — industrialized as automatic differentiation.
Representation learning
“Discover features, don't design them” became deep learning's defining idea — and the paper's shared receptive fields prefigured the CNN.
One idea — send the error backward — and machines began to discover their own ideas.