The lasting contribution here isn't gradient descent itself — versions of it predate this work — but the demonstration that a network can discover its own internal representations rather than relying on hand-crafted features. That reframing, more than the algorithm, is what revived neural networks as a serious research program after the perceptron critique had stalled them.
Key Findings
- The generalized delta rule propagates error backward through hidden layers, making multi-layer networks trainable end-to-end — the capability single-layer perceptrons fundamentally lacked.
- Hidden units learn distributed codes (detecting symmetry, solving XOR and parity) that no one hand-programmed, showing useful features can be learned rather than designed.
- Across the authors' simulations, training rarely gets trapped in debilitating local minima — the central theoretical worry about descending a non-convex error surface.
How It Works
The network minimizes squared output error by computing, for every weight, how a small change would affect that error, then adjusting along the negative gradient. The chain rule lets the same correction signal flow layer by layer from outputs back to inputs, so hidden weights with no explicit target still receive a learning signal.
Where It Fits
Great fit if you want the conceptual origin of how modern deep learning trains — read it for the ideas (representation learning, credit assignment) and the careful simulations. Look elsewhere if you need modern practice: nothing here on stochastic minibatches, ReLU, normalization, or scaling, which came decades later.
