Don't learn the mapping. Learn what to add to the input.
By 2015 the recipe for accuracy seemed obvious: stack more layers. A deeper network should never do worse than a shallow one — it could always copy the shallow net and set the extra layers to identity mappings (do-nothing layers that output exactly their input). Yet in practice, adding layers to a plain deep network made its training error rise. Not overfitting — the deeper model simply got harder to optimize. The authors call this the degradation problem.
Their fix reframes what a block of layers is asked to learn. Instead of
forcing a stack to fit some target mapping H(x) from
scratch, let it learn only the residual — the difference
F(x) = H(x) − x — then add the original input back, so the
block computes F(x) + x. A free identity shortcut
carries x straight across. If the best thing to do is
nothing, the block just pushes F(x) toward zero — far
easier than coaxing a pile of nonlinear layers into copying their own
input.
Adding the input back turns "do nothing" from the hardest thing for a deep stack to learn into the easiest.
The paradox
Deeper should be free — so why did it hurt?
Step 1 · The black box & the prize
A box that labels images — and won everything in 2015.
Here is the whole story in one picture. An image goes in, a label comes out. The box is a ResNet-152: a 152-layer convolutional network (a stack of learned image filters). This box will follow us through all eight steps.
In 2015 this box swept the field. An ensemble of these nets hit 3.57% top-5 error on the ImageNet test set — the fraction of images where the true label isn't among the model's top five guesses — and won 1st place at ILSVRC 2015 classification. At 152 layers it is 8× deeper than VGG, the previous champion, yet has lower computational complexity. The rest of this page explains how depth that deep was even made trainable.
From the abstract: “Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places.”
Step 2 · “Just stack more layers?”
A deeper net can always copy a shallower one.
You've seen the box win at 152 layers. Before we open it, sit with the clean intuition that made everyone expect depth to be free.
Take any trained shallow network. Build a deeper one by tacking extra layers on top — and make every extra layer an identity mapping, a do-nothing layer that passes its input straight through. The deeper net now computes exactly what the shallow one did. So a deeper model has a solution that is at least as good as the shallow one: in principle, adding layers should never raise training error.
Shallow net
A working network, some accuracy reached.
+ identity layers
Extra layers do nothing → same output, same accuracy.
The expectation
So depth should be a free lunch: deeper ≤ shallow error.
The existence of a do-nothing solution means a deeper net is never worse in principle. So if a deeper net trains worse in practice, the problem isn't capacity — it's that the optimizer can't find that solution. Keep this; Step 3 breaks it.
Step 3 · The degradation problem
In practice, deeper plain nets trained worse.
Step 2 said depth should be free. Reality disagreed — and the way it disagreed is the clue that cracks the whole problem.
Train a plain net — a straight stack of layers with no shortcuts, the old design. The authors found that beyond a point, adding layers raised training error, not just test error. On ImageNet, the 18-layer plain net beat the 34-layer plain net; on CIFAR-10, a 56-layer plain net trained worse than a 20-layer one.
Plain nets · deeper is worse
ImageNet top-1: plain-18 27.94 · plain-34 28.54. The deeper one is worse.
Residual nets · deeper is better
Same depths, with shortcuts: ResNet-18 27.88 · ResNet-34 25.03. The crossover is the fix (Step 6).
Two things make this a genuine puzzle, not a routine failure. First, it is training error that rises — so it is not overfitting (overfitting would lower training error while raising test error). The solver simply can't fit the deeper model as well, even on the data it can see.
Second, it is not vanishing gradients — the classic reason deep nets were once hard to train. These nets use batch normalization (BN), a per-layer rescaling that keeps the forward signal's spread healthy and the backward gradients non-vanishing; the authors verified the gradients stay alive. The capacity is there (Step 2) and the gradients are there — yet the optimizer still can't reach the do-nothing solution. That is the degradation problem.
A deep stack of nonlinear layers finds it hard to learn an identity mapping — the very solution Step 2 promised was available. If we could make "output = input" the easy default instead of a hard target, degradation would vanish. That is exactly what Part B does.
Figure 1 / Table 2 of the paper. Plain-vs-residual nets in Step 6 share identical depth, width, parameters and FLOPs — identity shortcuts are free — so this is a clean apples-to-apples comparison.
The fix
Learn the residual · a free shortcut · stack it deep
Step 4 · Learn the residual
Let the block correct the input, not replace it.
Step 3 left us needing "do nothing" to be easy. Here's the rewiring that delivers it — and before any formula, you should compute it by hand.
Take a tiny 3-number activation entering one block: x = [2.0, −1.0, 0.5]. Suppose the block's ideal output is almost the same as its input — it mostly needs to pass the signal through, with a small tweak. Watch how two designs handle that.
Plain block · learn the whole thing
Two nonlinear layers must reproduce ≈ [2.0, −1.0, 0.5] from scratch — the layers have to build the signal themselves. At depth, the solver struggles to even land on "copy the input." This is the degradation problem in miniature.
Residual block · learn only the change
The layers learn a small correction F(x) = [0.10, 0.00, −0.05], and a shortcut adds the input back: [2.10, −1.00, 0.45]. If the best move is nothing, drive F → 0 and the output equals x exactly.
Drag the residual size — watch output = F(x) + x update
‖F(x)‖ ≈ 0.11 — a small correction around the input.
Slide to zero and the residual strip empties: output = x exactly, for free. The block only ever learns the change, never the whole signal — so "do nothing" is now the trivial default, not the hard target Step 3 stumbled on.
Now the formula arrives as a receipt for what you just did, not as new magic. The block computes:
y=F(x, {Wi})+x
F = W₂ · σ(W₁x), σ = ReLU · then a final ReLU on the sum · F = your residual strip · x = the shortcut
The hypothesis: a residual is easier to learn than the original, unreferenced mapping. If an identity mapping were optimal, pushing the residual to zero is trivial; fitting an identity out of a stack of nonlinear layers (Step 3) is not. Most useful blocks turn out to need only a small F — a gentle perturbation around the input.
The numbers here are a toy 3-dim illustration. The real activations are full feature maps, and F is two or three conv layers — but the arithmetic "output = correction + input" is exactly this.
Step 5 · A shortcut that costs nothing
The skip connection is just an addition.
You added the input back by hand in Step 4. Here's the wiring that does it — the shortcut connection (or skip connection): a wire that skips one or more layers and carries the input forward unchanged.
The shortcut performs an identity mapping: it copies
x across and adds it, element-wise, to the stacked
layers' output, then a final ReLU runs. That addition adds
no extra parameters and no extra computation — it's free.
The whole network still trains end-to-end with ordinary SGD and
backpropagation; nothing about the optimizer changes.
One wrinkle: the addition needs both sides to be the same shape. When a block changes dimensions (more channels, or a smaller feature map), the paper offers two options:
| Option | What the shortcut does | Cost |
|---|---|---|
| A · zero-pad | Still identity, with extra zeros padding the new channels | 0 params (free) |
| B · projection | A 1×1 conv Ws reshapes x: y = F(x) + Wsx | a few params |
Identity shortcuts (A) are enough and the most economical — they keep even the deepest, bottleneck-based nets efficient. Projections (B) are used only where dimensions actually change.
Shortcut / skip connection: a wire that skips layers and adds the input forward. Identity mapping: output = input. Both terms recur below — the shortcut is what carries the identity.
Step 6 · Stack it — plain vs residual at depth
With the shortcut, depth helps again.
You have one free shortcut (Step 5). Now stack the blocks deep and re-run the exact experiment that broke plain nets in Step 3 — same depths, same width, same parameter count.
The result flips. Where plain-34 was worse than plain-18, ResNet-34 beats ResNet-18 — and it beats plain-34 outright. The degradation is gone: extra depth now lowers training error, exactly as Step 2 promised it should.
| Network | 18-layer | 34-layer | Depth verdict |
|---|---|---|---|
| Plain net | 27.94 | 28.54 | deeper is worse ✗ |
| ResNet | 27.88 | 25.03 | deeper is better ✓ |
Read the table by column and the fix is obvious. Going 18 → 34 adds 0.6 error to the plain net but removes ~2.85 error from the residual net. The identity shortcuts didn't add capacity — recall they're free — they made the existing capacity reachable by the optimizer.
Same layers, same parameters, same gradients — one addition per block, and depth stops hurting. The paper's own framing: residual learning eases optimization, and these nets "enjoy accuracy gains from greatly increased depth."
The residual panel of Step 3's chart is this story drawn as a curve: the 34-layer line crosses below the 18-layer line. Use Table 2's numbers throughout (10-crop and other settings differ slightly).
Step 7 · Going very deep — the bottleneck
A cheaper block buys 50, 101, 152 layers.
34 layers works (Step 6). To go far deeper without the compute exploding, the deepest ResNets swap the basic two-3×3 block for a bottleneck block.
The bottleneck is three layers: a 1×1 conv that
shrinks the channel count, a 3×3 conv that does the
real work at that small width, and a 1×1 conv that
restores the channels for the addition. The expensive 3×3 now
operates on far fewer channels, so a three-layer bottleneck costs about
the same as a two-layer basic block — while adding more depth.
Same block, scaled to any depth. ResNet-18/34 use the basic two-3×3 block; ResNet-50/101/152 use bottlenecks. The payoff is striking — even at 152 layers, the network has lower compute than VGG:
FLOPs = floating-point operations, a measure of compute per image. BN sits after every convolution and before activation; the nets use no dropout, are initialized from scratch (He init), and trained with SGD.
The payoff
Did it work — and can you now explain it?
Step 8 · Results & recap
Did it work? — and can you now explain it?
You've built the idea: residual reframe (Step 4), free shortcut (Step 5), depth that helps (Step 6), bottleneck for the deepest nets (Step 7). Final question — how far did it actually go?
Report card · ImageNet single-model top-5 error %
ResNet-152 · single model
ResNet-101 · single model
ResNet-50 · single model
VGG-16 · single model
GoogLeNet · single model
0top-5 error % · lower is better10
About that 1202-layer net: it still optimizes fine — no optimization difficulty — but lands slightly worse (7.93%) than the 110-layer one. The authors attribute that to overfitting the small CIFAR dataset, not degradation. The residual fix removed the optimization wall; depth itself is no longer the limit.
Now you can explain it. Five questions — answer each out loud before opening it. If all five come easily, you've genuinely got this paper.
Why do deep PLAIN nets get worse — even on training data?
It's the degradation problem: training error rises with depth, so it's not overfitting. It's also not vanishing gradients (BN keeps them healthy). A deep stack of nonlinear layers simply struggles to even fit an identity mapping — the optimizer can't reach the do-nothing solution that Step 2 proved exists.
What does a residual block actually learn?
Only the residual F(x) = H(x) − x — the change to make to its input — then a shortcut adds x back, so the block outputs F(x) + x. You computed it in Step 4: input [2.0, −1.0, 0.5] plus a small F = [0.10, 0.00, −0.05] gives [2.10, −1.00, 0.45].
Why is the identity shortcut "free"?
It's just an element-wise addition of x to the layers' output — no weights, no extra FLOPs, and the net still trains with plain SGD. Only when dimensions change does a 1×1 projection add a few parameters; everywhere else the shortcut is pure identity.
Why a bottleneck block for 50+ layers?
A 1×1 conv shrinks the channels, the 3×3 does its work at that smaller width, and a 1×1 restores them. The costly 3×3 runs on far fewer channels, so a three-layer bottleneck costs about the same as a basic block — letting 50/101/152-layer nets stay cheaper than VGG.
If it's so deep, why didn't it just overfit?
On ImageNet, deeper ResNets actually generalized better, with BN and heavy augmentation regularizing them. Overfitting only showed up at the extreme 1202-layer net on tiny CIFAR-10 — and even there the optimization was fine. The fix targets optimization, not capacity.
What happened next — the residual / skip connection became one of the most universal ideas in deep learning. It is why the Transformer wraps every sub-layer in a residual connection, and it underpins DenseNet, U-Net, and essentially every very deep network since. Training networks hundreds of layers deep went from impossible to routine.
“We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions.” — the abstract, He et al., 2015
The skip connection is everywhere now.
Residual learning didn't stay in image recognition. The simple move — add the input back — turned out to be one of the most generic, reusable ideas in all of deep learning, quietly sitting inside nearly every deep model built since.
Transformers
Every sub-layer is wrapped in a residual connection — LayerNorm(x + Sublayer(x)). The idea you just learned is a big part of why attention could be stacked deep enough to power modern LLMs.
DenseNet · U-Net · gradient highways
Skip connections became a default building block across vision and beyond — feature reuse, encoder-decoder bridges, and a clean path for gradients straight back through very deep nets.
Depth, unlocked
Training networks hundreds of layers deep stopped being a research stunt and became routine — across vision, speech, science and multimodal models, depth is now a dial you can turn.
One addition per block — and "make it deeper" became something you could simply do.