Keep the shortcut a clean identity, and the network becomes a highway.
The first ResNet paper showed that adding the input back —
F(x) + x — let networks go very deep. This sequel asks the
obvious next question: why, and exactly how clean does
that + x path have to be? The answer is a piece of
algebra. If the skip connection is a plain identity mapping (it
copies the input across untouched) and the activation after the
addition is also identity, then the value at any deep layer becomes the
earlier input plus a sum of residuals, and — by the same token —
its gradient becomes 1 plus a sum of terms.
That additive 1 is the whole story. It is a direct wire
from the loss back to every layer that never multiplies the gradient
down to zero, no matter how deep you stack. So the rule the authors
extract is almost a warning: anything you put on that path — a
0.5 scale, a gate, a 1×1 conv, a ReLU — multiplies into that highway and
can choke it. The cleanest path, doing literally nothing, wins. Their
pre-activation unit (move BN and ReLU before the convs)
is simply the rewiring that keeps both the shortcut and the
post-addition step perfectly clean.
The shortcut isn't a helpful extra. It's a protected lane for the signal and the gradient — and every brick you lay in that lane slows them both down.
The clean highway
ResNet works — but why? Follow one signal up, one gradient down.
Step 1 · The black box & the open question
ResNet stacks one block — a residual unit — hundreds of times.
Start with what the first ResNet paper gave us. A deep residual network
is one small block, the residual unit, repeated over and over.
Each unit takes a feature vector x, sends it through a
couple of weight layers to compute a residual
F(x) (the small correction those layers learn), and adds the
untouched input back through a shortcut — a wire that
skips the layers. That sum is the input to the next unit. This stack is
our running example for the whole page; we'll trace one signal up it and
one gradient back down it.
The original paper proved this works — depth stopped hurting.
But it left the mechanism vague: the shortcut was sometimes a plain
identity, sometimes a projection; an extra ReLU sat after the addition.
This sequel pins down the one thing that matters: how clean that
+ x path is. Keep it a pure identity and you get the
tidy formula above; touch it and the formula — and the training —
breaks.
From the abstract: “… identity mappings as the skip connections and after-addition activation … make training easy and improve generalization.” Residual unit, shortcut, residual F(x): all defined here and reused below.
Step 2 · The forward signal
Unroll the stack and the signal is just input + a sum.
Step 1 drew the tower. Now let's do the algebra on one unit, then unroll it — and watch the whole deep net collapse into one clean line.
Write a residual unit precisely. It does two things:
it forms y = h(x) + F(x), where h is the
shortcut, then it applies a final activation f to get the
next input, xnext = f(y). The paper's claim has
a precondition: make both h and f the
identity — h(x) = x and f(y) = y.
Then a unit is simply xℓ+1 = xℓ + F(xℓ).
One unit, in general
y = h(x) + F(x), then xℓ₊₁ = f(y).
Make h and f identity
No clipping, no scaling — a plain add.
Unroll L − ℓ units
Any deep feature = an earlier one + residuals.
xL=xℓ+ΣF(xi)
sum over i = ℓ … L−1 · holds for any deeper unit L and any earlier unit ℓ — a feature is its ancestor plus the residuals between them
There is no chain of matrix multiplies between xℓ
and xL — only additions. The signal from layer ℓ reaches
layer L directly, undiluted by everything stacked in between. This
additive, multiply-free path is the property the rest of the page protects.
Step 3 shows it does the same favor for the gradient.
Equations 1–4 of the paper. The precondition matters: this clean form only holds when h and f are identity. Replace either one and the additions turn back into multiplications — which is exactly Part B's warning.
Step 3 · The gradient highway
The same add gives the gradient an unkillable "+1".
Step 2 sent one signal up the highway. Training, though, runs the other way: the gradient flows down. Differentiate that clean forward formula and the highway reappears — this time protecting learning itself.
Apply the chain rule to xL = xℓ + Σ F.
The derivative of xL with respect to the earlier
xℓ splits into two pieces: the derivative of the
xℓ term, which is just 1, plus the
derivative of the sum of residuals. So the gradient that reaches layer ℓ
is the loss-gradient at the top, multiplied by (1 + something).
∂E/∂xℓ=∂E/∂xL·(1+∂/∂xℓ Σ F)
the 1 wires the top gradient straight to layer ℓ · the Σ F term is what the weight layers learn from · their sum, not their product, is what saves you
Two consequences fall out, and both are the point of the paper. First,
the 1 guarantees the top-level gradient reaches every
layer intact — it is added in, never multiplied away. Second, the
other term is a sum over a mini-batch; it would have to be exactly
−1 for every single example at once to cancel the 1,
which essentially never happens. So the gradient cannot vanish,
however deep the net. Compare that with a plain net, where the gradient is
a long product of layer terms — shrink each a little and the
product collapses toward zero.
Scale the shortcut λ and add depth — watch the gradient survive or die
λ = 1.00 → gradient factor ≈ 1.00 at 50 units — the identity highway keeps it alive.
Leave λ at 1 and the curve hugs the top no matter how deep — the
+1 path. Drop λ below 1 and the shortcut becomes a
multiply: the factor is λ(L−ℓ), which
decays toward zero as depth grows. At λ = 0.5 over 50 units the gradient
is essentially gone. That is what putting anything on the
shortcut does.
A pure identity shortcut keeps the gradient an addition; anything else turns it into a multiplication that compounds with depth. The forward signal (Step 2) and the backward gradient (here) are protected by the exact same clean path. Part B is one long demonstration that this is not optional.
Equation 5 of the paper. The plain-net gradient is a product Πᵢ (∂xᵢ₊₁/∂xᵢ); with small factors it underflows. The residual gradient is 1 + (a sum), so it doesn't. The λ-decay curve is the case the paper analyses in Eq. 7 (constant-scaling shortcut).
Keep it clean
Touch the path → it breaks · the fix is one reorder of BN and ReLU
Step 4 · Don't touch the shortcut
Every "improvement" on the skip path made it worse.
Step 3 said the highway only works if the shortcut is a pure identity. That sounds like a slogan — so the authors tested it the hard way, by trying to "improve" the shortcut five different ways and measuring what happens.
All five experiments use the same ResNet-110 on CIFAR-10. The
only thing that changes is what the shortcut h(x) does
before the addition. The plain identity baseline scores 6.61%
error. Watch every alternative lose to it.
| What the shortcut does | h(x) = | Error % |
|---|---|---|
| Identity (do nothing) | x | 6.61 |
| Constant scaling | 0.5 · x | 12.35 |
| Exclusive gating | (1−g) · x | 8.70 |
| Shortcut-only gating | g · x | 6.91 |
| 1×1 conv shortcut | Ws · x | 12.22 |
| Dropout on shortcut | drop(x) | fails |
A 1×1 conv shortcut has more parameters and more representational power than plain identity — yet it does worse (12.22% vs 6.61%). Capacity wasn't the bottleneck; a clean propagation path was. For a community trained to "add capacity," that is the surprise: on the shortcut, less is more.
Table 1. The 0.5 scaling and 1×1 conv hurt most because they attenuate or re-mix the signal on the very path that was supposed to stay clean; with 1000+ layers, even a 0.5 factor underflows the gradient (Step 3's λ curve).
Step 5 · The other identity — the activation after the add
The ReLU sitting on the addition quietly clips the highway.
Step 4 protected the shortcut h. But Step 2's clean formula
needed two things to be identity — and the second, the activation
f applied after the addition, is the one the original
ResNet quietly got wrong.
In the original unit, the order is conv → BN → ReLU → conv → BN, then the
shortcut is added, and then a final ReLU runs on the sum. That last
ReLU is f. It zeroes any negative value — so f is
not identity, and the sum that should flow untouched to the next unit
gets clipped. The deeper the net, the more often that clipping lands on the
highway.
f. Because ReLU is not identity, the “add” the next unit receives has already been truncated. The highway has a gate on it.
One clipped value is harmless. But the post-addition ReLU sits on the
recursion between every pair of units — so its effect compounds
exactly like the λ-multiply in Step 3. To restore Step 2's clean formula,
we need to move the activation off the addition and make
f a true identity. That is the pre-activation idea.
Activation f: the function applied to y = h(x) + F(x) before it becomes the next unit's input. In the original design f = ReLU; the goal is f = identity. h was handled in Step 4; f is handled next.
Step 6 · Pre-activation — the v2 unit
Move BN and ReLU before the convs; the add stays clean.
Step 5 found the culprit: the ReLU on the addition. The fix is almost too simple — reorder the same operations so the activation happens before the weight layers, not after the add. Same parts, new order.
In the pre-activation unit the order becomes BN → ReLU → conv →
BN → ReLU → conv, and the result is added to the shortcut with nothing
after it. The addition is now the unit's exact output —
f is the identity. Every ReLU still runs; it just runs on the
way into the convs, where clipping is normal and harmless, never
on the highway. Compute it by hand and the difference is stark.
Flip the activation placement — see whether the add survives to the next unit
Pre-activation: f = identity — the sum [0.70, −0.70, 0.50] reaches the next unit untouched. The −0.70 survives.
Switch to "Original" and a ReLU runs on the sum: the
−0.70 is clipped to 0 (it turns coral). The
highway value that should have flowed on is destroyed — and this happens
at every unit. Pre-activation leaves the sum alone, so the
negative value survives to the next layer. Same arithmetic; one of them
keeps the highway open.
h and f are now identity — Step 2's clean formula holds exactly.
“First, the optimization is further eased … because f is an
identity mapping.” The clean highway is restored end to end. “Second,
using BN as pre-activation improves regularization” — every conv now sees
normalized inputs, so the model also generalizes better. One
reorder; two wins.
Figure 4 / Section 4.1. Toy numbers above are a 3-dim illustration; real activations are full feature maps. The arithmetic — “add, then maybe clip” vs “add, done” — is exactly what the post-addition ReLU controls.
Step 7 · Did the reorder actually win?
Walking the ReLU off the add steadily lowers error.
Step 6 argued pre-activation by hand. The paper proves it by trying every
half-measure in between — moving the activation one step at a time and
measuring each. The trend is monotone: the closer f gets to
identity, the lower the error.
| Where the activation sits | ResNet-110 | ResNet-164 |
|---|---|---|
| Original · ReLU after add (f = ReLU) | 6.61 | 5.93 |
| BN after addition | 8.17 | 6.50 |
| ReLU before addition | 7.84 | 6.14 |
| ReLU-only pre-activation | 6.71 | 5.91 |
| Full pre-activation (BN-ReLU before conv) | 6.37 | 5.46 |
Two readings matter. First, naive tweaks hurt: shoving BN after
the addition, or forcing the residual non-negative with a ReLU before it,
both raise error — they put new obstacles on the path. Only the move that
makes f a genuine identity, full pre-activation, beats
the baseline. Second, the gap widens with depth: at 164 layers,
5.93 → 5.46. The deeper the net, the more a clean path is worth — exactly
what Step 3's compounding predicted.
f identity.Table 2. ResNet-164 here uses bottleneck units; ResNet-110 uses basic two-layer units. BN as pre-activation also regularizes, which is why pre-activation often helps test error even when training error is similar.
The payoff
A 1001-layer net that trains — and what you can now explain
Step 8 · Results — depth that finally pays off
With a clean path, 1001 layers beat 164.
You've protected the shortcut (Step 4), fixed the post-add activation (Steps 5–7). Now stack it to absurd depth — the place the original unit broke down — and see what the clean path buys.
The headline test is ResNet-1001 on CIFAR-10. With the original unit it overfits and stalls at 7.61%; with pre-activation it trains cleanly to 4.92% — and, tuned, to 4.62%. The deeper 1001-layer net now beats the 164-layer net (5.46%), the opposite of degradation.
Report card · CIFAR-10 test error % · pre-activation (v2)
ResNet-1001 · pre-activation
ResNet-164 · pre-activation
ResNet-1001 · original unit
ResNet-164 · original unit
0CIFAR-10 error % · lower is better10
Notice what changed between the two ResNet-1001 rows: same depth, same parameters, same data. The only difference is whether the shortcut and the post-addition step are clean identities. That 7.61 → 4.62 gap is the entire thesis of the paper, measured.
Tables 2–5. ImageNet uses 200-layer nets on a 320×320 crop; CIFAR uses the 1001- and 164-layer nets. The original ResNet-200 actually overfits before pre-activation’s regularization is added — depth without a clean path is fragile.
Step 9 · Can you now explain it?
Five questions — answer each before you open it.
You built the whole argument: the clean forward sum (Step 2), the never-vanishing gradient (Step 3), why the shortcut (Step 4) and the post-add activation (Steps 5–7) must both be identity, and the numbers (Step 8). Now prove it to yourself.
Answer each out loud first. If all five come easily, you've genuinely got this paper.
Why does a pure identity shortcut keep gradients alive?
Differentiate xL = xℓ + Σ F: the gradient at layer ℓ is the top gradient times (1 + a sum). That additive 1 is a direct wire — the gradient is added, never multiplied down. The sum would need to be exactly −1 for every example to cancel it, which never happens. So it cannot vanish, however deep.
What goes wrong if you scale the shortcut by 0.5?
The shortcut stops being identity and becomes a multiply. Over L − ℓ units the gradient factor is 0.5(L−ℓ), which underflows to ~0 at great depth. Measured: ResNet-110 goes 6.61% → 12.35%. You saw this as the decaying curve in Step 3.
What was wrong with the original post-addition ReLU?
That ReLU is the activation f. Because ReLU clips negatives, f is not an identity mapping, so the sum that should flow untouched to the next unit gets truncated — at every unit. The clean forward formula (Step 2) no longer holds.
What exactly does "pre-activation" change?
It reorders each unit to BN → ReLU → conv → BN → ReLU → conv, adds the shortcut, and puts nothing after the add. So f becomes identity and the highway is clean again. Bonus: BN now normalizes every conv's input, which also improves regularization.
Why is "add capacity" the wrong instinct here?
A 1×1 conv shortcut has more parameters yet does worse (12.22% vs 6.61%). The bottleneck was never representational power — it was a clean propagation path. On the shortcut, the best thing you can do is nothing.
What happened next — “keep the residual path clean” outlived image recognition. It became the design rule for the Transformer’s residual stream and for every very deep stack since: don’t put learned transforms on the skip; let the signal and the gradient run straight through.
“If both h and f are identity mappings, the signal could be directly propagated from one unit to any other unit, in both forward and backward passes.” — the paper, He et al., 2016
“Keep the residual path clean” became a law of deep learning.
This paper turned a working trick into a principle: the shortcut is a protected lane for signal and gradient, and you put nothing learned on it. That rule, plus the pre-activation unit, propagated far beyond vision.
Transformers
Every sub-layer adds to a clean residual stream — x + Sublayer(x). Pre-norm Transformers (LayerNorm before the sublayer, nothing on the skip) are the direct descendant of pre-activation, and are why GPT-scale stacks train at all.
ResNet-v2 everywhere
The pre-activation unit became the standard residual block shipped by modern frameworks, and seeded DenseNet, U-Net and the highway-style designs that followed — feature reuse and gradients running straight back.
1000-layer training
Training nets a thousand layers deep stopped being a stunt. Once the path is a clean identity, depth is no longer a battle against vanishing gradients — it's just a number you can turn up.
Less on the shortcut, more from the depth — the cleanest path, doing nothing, won.