The original ResNet paper showed that residual connections let you train absurdly deep networks; this follow-up answers the question it left open — why. The insight is almost embarrassingly simple: if the skip connection and the post-addition activation are both kept as clean identity mappings, then the signal at any layer becomes the input plus a sum of residuals, and the gradient at any layer becomes 1 plus a sum of terms. That additive "1" never lets the gradient vanish, no matter how deep you stack. Depth stops being a battle against degradation and becomes nearly free.
Key Findings
- The skip path must stay pristine. The authors ablate scaling, gating, and 1x1 convolutions on the shortcut — every modification that touches the identity path hurts optimization. The cleanest path, doing literally nothing, wins. That is a counterintuitive result for an architecture community trained to add capacity.
- Pre-activation reorders the block. Moving BatchNorm and ReLU before the convolutions (rather than after the addition) keeps the addition output unmodified and turns the activation into a true identity-preserving function. This is the "ResNet-v2" unit that later networks adopted.
- It unlocks extreme depth. With these two changes, a 1001-layer ResNet trains stably and reaches 4.62% error on CIFAR-10, with gains on CIFAR-100 and a 200-layer ImageNet model — depths that diverged under the original formulation.
- Gradient flow, not representation, was the real bottleneck. The paper reframes deep-network training as a propagation problem solved by architecture, not by better optimizers.
Why It Still Matters / When to Skip
Great fit if you want to understand mechanistically why residual connections work, or trace where the now-universal pre-activation pattern and "keep the residual stream clean" principle — later echoed in Transformers — came from. Look elsewhere if you only need to use ResNets in practice: any modern framework ships v2 blocks by default, and you do not need the propagation algebra to call them. This is the explanatory paper, not a tutorial.
