The counterintuitive problem this paper named was not vanishing gradients but degradation: adding layers to an already-deep network made training error go up, even though a deeper net can in principle just copy the shallower one. That observation reframes the whole contribution. The fix is almost embarrassingly simple — let each block learn the residual F(x) = H(x) - x and add the input back through a shortcut — but it turned "depth" from a liability into the main lever for accuracy.
Key Findings
- Degradation, not overfitting, was the wall. Plain deep nets had higher training error than shallower ones; residual reformulation removed that ceiling, so accuracy now climbed with depth instead of collapsing.
- Identity shortcuts cost nothing. The skip connection adds no parameters and negligible compute, yet makes optimization dramatically easier — a 152-layer ResNet is 8x deeper than VGG while having lower complexity.
- The numbers held up. An ensemble reached 3.57% top-5 error on ImageNet, winning ILSVRC 2015, and the same backbone drove a 28% relative gain on COCO detection.
- Depth as a transferable feature. Because representations got deeper and trainable, ResNet became the default backbone far beyond classification — detection, segmentation, and eventually non-vision tasks.
How It Works
A residual block computes y = F(x) + x, so the layers only need to model the difference from the input. When the optimal mapping is close to identity, driving F toward zero is far easier than fitting an unreferenced function from scratch — which is precisely the regime very deep stacks fall into.
Who It Fits / When to Skip
Great fit if you want the origin of skip connections — the idea now baked into Transformers, U-Nets, and almost every deep architecture — or to understand why depth stopped being dangerous. Look elsewhere if you need a current vision recipe: attention-based backbones (ViT) and modern training tricks came later, and this paper is the structural foundation, not a leaderboard result.
