Dropout and LSTMs aren't incompatible. You were just putting the noise in the wrong place.
Dropout — randomly zeroing units during training so the network can't lean on fragile co-adaptations — was the most powerful regularizer of the era for feed-forward nets, but it made recurrent nets worse. The accepted wisdom was that recurrence simply amplifies the noise. So in practice people used RNNs that were too small, because bigger ones overfit.
The authors' bet is almost embarrassingly small: an LSTM has two kinds of connection — vertical ones that pass information up from the layer below at the same timestep, and recurrent ones that carry memory forward to the next timestep. Spray dropout on everything and the recurrent path gets corrupted at every timestep, so the cell can never hold a thought for long. Confine dropout to the vertical connections, leave the recurrent path untouched, and the contradiction disappears.
“The main idea is to apply the dropout operator only to the non-recurrent connections.” — Zaremba, Sutskever & Vinyals
The contradiction
Why dropout — the best regularizer of its day — broke recurrent nets.
Step 1 · The puzzle
The best regularizer of its era made recurrent nets worse, not better.
Here is the whole puzzle in one picture. Take a recurrent net, switch on plain dropout — the trick that was making every feed-forward network generalize better — and the error goes up. That is the mystery this paper solves:
On Penn Tree Bank — the standard word-level language-modeling benchmark — a non-regularized LSTM scored 114.5 test perplexity (lower is better). Plain dropout couldn't help; the only way to keep a recurrent net from overfitting was to keep it small. So people did, and capped their models' size.
With dropout applied correctly — only to the non-recurrent
connections — the same kind of LSTM, now much larger, reached
78.4. The rest of this page is about that word “correctly”: what
the two kinds of connection are, why one of them must stay noiseless,
and how a single dropout operator D in one equation makes
all the difference.
From the abstract: “Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks.”
Step 2 · The task, made concrete
Guess the next word, over and over — and we grade you by how surprised you are.
You've seen the puzzle: dropout helps everywhere except recurrent nets. Before opening the LSTM, pin down the task the paper measures everything on — and the one number that decides who wins.
The headline benchmark is language modeling on the Penn Tree Bank (PTB) — a corpus of Wall Street Journal text: 929k training words, 73k for validation, 82k for test, a 10,000-word vocabulary. The job is simple to state: read the words so far, and put a probability on every possible next word.
The score is perplexity — roughly, “how many words was the model effectively choosing between?” If it's perfectly sure of the right word every time, perplexity is 1; if it's as lost as random guessing over the whole vocabulary, perplexity is 10,000. Lower is better. Our running sentence below has the model predict the last word:
Step 3 · The shape of the machine
Unroll the LSTM and you get a grid: time runs sideways, depth runs up.
You know the task: predict each next word. Here's the picture the rest of the page zooms into — what a deep LSTM actually looks like once you unroll it across the sentence.
An LSTM (long short-term memory) is a recurrent cell that reads one word at a time and keeps a running memory. Stack two of them and you get a 2-layer deep LSTM. Unrolling just means drawing one copy of the cell per timestep — so the same cell, reused, becomes a row of cells reading the sentence left to right.
Drawn this way, two kinds of arrow appear, and the whole paper hinges on telling them apart:
The whole paper is a claim about which arrows dropout may touch. Vertical arrows can take noise. Horizontal arrows — the memory path — must stay clean. Keep this grid in mind: every step from here on points back to it.
The PTB LSTMs in the paper are exactly this shape: 2 layers, unrolled for 35 timesteps. The “medium” model has 650 units per layer; the “large” model 1500. Everything in Parts B and C happens inside this grid.
The one-line fix
Where dropout goes · the L+1 corruption count · why the recurrent path must stay clean
Step 4 · Toggle the dropout, see what breaks
Put the noise on the vertical wires and it helps; put it on the memory wires and you wipe the memory.
You have the grid from Step 3: vertical wires carry a word up the layers, horizontal wires carry memory forward in time. This is the heart of the paper — so before any equation, do it by hand.
Dropout is an operator D(·) that, each training
step, randomly zeroes some of the values flowing along a connection.
The only design choice is which connections it touches. Below is
our unrolled LSTM reading “the cat … was hungry.” Flip the switch and
watch which wires get the dashed dropout noise — and what it does to
the memory of the subject “cat.”
Choose where dropout goes — watch which wires get corrupted
Dropout's dashed coral noise = wires being randomly zeroed during training. On the vertical wires it just forces robust per-word features. On the horizontal wires it cuts the memory of “cat” a little at every timestep — so by the time you reach “hungry?”, what the subject was is gone.
See the one symbol that changes in the LSTM equation
The gates of an LSTM are computed from two inputs: the value coming
up from the layer below, hl−1t
(vertical), and the value coming across from the previous
timestep, hlt−1 (recurrent). The fix
is to wrap only the first one in the dropout operator:
plain : gates = T( hl−1t , hlt−1 )
fixed : gates = T( D(hl−1t) , hlt−1 )
That is the whole contribution: the recurrent term
hlt−1 is left bare. (Full LSTM gate
equations are in Step 7.)
Same idea, said plainly: dropout may corrupt the signal as it moves between layers, but never as it moves between timesteps. Next we count exactly how much corruption a memory survives — and find a number that doesn't depend on how far back the memory came from.
Step 5 · Count the corruption — the L+1 rule
A memory crosses dropout exactly L+1 times — no matter how far back it came from.
In Step 4 you saw where the noise goes. Now the punchline: keeping the recurrent path clean means a memory survives a fixed, small amount of corruption — and you can literally count it.
Trace the thick path a memory takes to travel from an early word to a
later prediction. It enters at the bottom, rides the
clean recurrent rails sideways across
however many timesteps it needs, then climbs the
vertical wires to the output. Dropout
only sits on the vertical wires — so the memory crosses it once per
layer plus once at the output: L + 1 times, where
L is the depth. Drag the distance and watch that count
refuse to grow.
Drag the distance — the corruption count never changes
The coral ✗ marks are the only places dropout touches the memory. Move the slider from 1 timestep to 7 and the path gets longer sideways — but it still crosses dropout exactly L + 1 = 3 times (2 layers + 1 output). Plain dropout, by contrast, would add a cross on every sideways hop, so a memory 7 steps old would be corrupted ~10 times.
The corruption a memory accumulates is independent of the number of timesteps it travels. That is exactly what lets an LSTM still “remember events that occurred many timesteps in the past,” even while dropout regularizes everything else. Constant corruption, unbounded memory.
The paper's Figure 3 draws this same thick path and says it plainly: “the information is corrupted by the dropout operator exactly L + 1 times, and this number is independent of the number of timesteps traversed.”
Step 6 · Why the recurrent path is so fragile
Noise on the memory wire compounds with every word — long thoughts never make it.
The L+1 rule was the good case. Now see the bad case it avoids: what happens to a memory if you do drop on the recurrent wires — the choice everyone tried first.
On the recurrent path, a memory passes through the same dropout point once for every timestep it survives. Each pass zeroes some of it; the damage accumulates with sequence length. A thought that needs to last 30 words gets hit ~30 times — so it simply can't. That is precisely why earlier attempts found that “the recurrence amplifies noise, which in turn hurts learning.”
Both prior camps were half-right. Dropout does get amplified by recurrence — but only on the recurrent connections. Move it off them and the amplification is gone, while every other connection still enjoys the regularization. That single distinction is the paper.
Step 7 · Now the equations — the relief
Everything you just did by hand is one wrapped term in one line.
You've toggled the placement, counted the L+1 crossings, and seen why the recurrent path must stay clean. Here are the actual equations — and because you already have the intuition, they should read as a summary, not a wall.
An LSTM layer computes four gates from its two inputs — the value from
below hl−1t and the value from before
hlt−1 — then updates a memory cell
c and emits a hidden state h:
(i, f, o, g) = (sigm, sigm, sigm, tanh) · T2n,4n(hl−1t, hlt−1)
clt = f ⊙ clt−1 + i ⊙ g · hlt = o ⊙ tanh(clt)
i = input gate · f = forget gate · o = output gate · g = candidate · ⊙ = elementwise product · T = affine map
The entire paper changes exactly one thing in the line above —
it wraps the from-below input, and only that input, in the dropout
operator D:
(i, f, o, g) = (sigm, sigm, sigm, tanh) · T2n,4n(D(hl−1t), hlt−1)
D zeroes a random subset of its argument · note hlt−1 (the recurrent term) is left bare
Nothing here is LSTM-specific. The same recipe — dropout on the non-recurrent connections, none on the recurrent ones — applies to any recurrent architecture. The authors note it “is likely to work well with other RNN architectures as well,” and indeed it does.
That is the complete method. No new layer, no new loss, no new optimizer — one dropout operator, placed deliberately. Part C is what it bought.
The payoff
PTB perplexity · bigger nets become trainable · it generalized everywhere
Step 8 · Did it work? The PTB report card
114.5 → 82.7 → 78.4 — and an ensemble pushed it to 68.7.
The method is complete: dropout on the vertical wires only. Now the numbers. Everything below is word-level perplexity on Penn Tree Bank — lower is better, so shorter bars win.
The non-regularized LSTM had to stay small (only 200 units per layer) or it overfit; it scored 114.5 test perplexity. Once dropout kept overfitting in check, the authors scaled up: a “medium” LSTM (650 units) hit 82.7, and a “large” one (1500 units) hit 78.4 — a single model beating every prior single model. Averaging 38 large models reached 68.7.
Report card · PTB word-level test perplexity (shorter = better)
Non-regularized LSTM · small (200u)
Regularized LSTM · medium (650u)
Regularized LSTM · large (1500u)
Ensemble · 38 large regularized LSTMs
0perplexity →~120
All numbers from Table 1. Validation/test pairs: non-reg 120.7 / 114.5 · medium 86.2 / 82.7 · large 82.2 / 78.4 · 38-model ensemble 71.9 / 68.7. Each LSTM is the 2-layer, 35-step shape from Step 3.
Step 9 · The two recipes — and what dropout does each step
Medium uses 50% dropout, large uses 65% — bigger model, more noise.
The report card showed the result; here's what actually produced it. Two configurations, identical in shape (2 layers, 35 steps), differing mainly in width and in how much dropout they take.
Notice the pattern: the larger the model, the more dropout it can use — 50% for medium, 65% for large. A bigger network has more capacity to overfit, so it wants stronger noise. The trick is that all of that noise lives on the vertical wires; the recurrent memory stays clean in both.
650 × 2 · 50% drop
2 layers, 650 units. 50% dropout on non-recurrent connections. 39 epochs, lr 1.0 then ÷1.2 each epoch after epoch 6. Grad-norm clip 5. → test ppl 82.7.
1500 × 2 · 65% drop
2 layers, 1500 units. 65% dropout on non-recurrent connections. 55 epochs, lr 1.0 then ÷1.15 after epoch 14. Grad-norm clip 10. → test ppl 78.4.
200 × 2 · no drop
Kept small on purpose — larger non-regularized nets overfit. 13 epochs, unrolled 20 steps. → test ppl 114.5, the model to beat.
Resample — each click draws a fresh dropout mask on the vertical wires
Large LSTM · 65% dropout on non-recurrent units
6 of 16 units kept this step.
Each training step zeros each unit on a vertical connection with probability 0.65, so every step trains a different thinned sub-network — forcing robust per-word features. The recurrent memory wires are never touched. At test time all units are used, their outputs scaled down to compensate.
There is no exotic optimizer, no auxiliary loss, no architecture change. Plain SGD, gradient clipping, a step-decayed learning rate — and one well-placed dropout. The whole gain comes from where the noise lives.
Step 10 · It generalized everywhere
Same rule, three more domains — and it helped in every one.
Language modeling was the headline. But if the fix were a PTB quirk it wouldn't matter much. The authors tried the exact same rule on three unrelated sequence tasks — a good sign it addresses something structural about recurrence, not one dataset.
Speech recognition, machine translation, and image caption generation are all sequence problems an LSTM can model. In each, dropout-on-non-recurrent-connections improved the held-out metric:
| Task · metric | No dropout | With dropout |
|---|---|---|
| Speech · frame accuracy (Icelandic, 93k utterances) | 68.9% | 70.5% |
| Translation · BLEU (WMT'14 En→Fr) | 25.9 | 29.03 |
| Translation · test perplexity (WMT'14 En→Fr) | 5.8 | 5.0 |
| Captioning · BLEU (MSCOCO) | 23.5 | 24.3 |
| Captioning · test perplexity (MSCOCO) | 8.47 | 7.99 |
Each task wanted a different dropout strength — 0.2 for the 4-layer translation LSTM, higher for the small overfit-prone speech and PTB models. The placement rule is universal; the amount is tuned to how much a given model would otherwise overfit. For captions, a single dropout model matched a 10-model ensemble.
Numbers from Tables 2–4. Speech is measured by frame accuracy (it correlates with word error rate but is cheaper to compute); higher is better. For BLEU and accuracy higher is better; for perplexity lower is better.
Step 11 · Now you can explain it
Our cat finally gets predicted — because its memory survived the journey.
The method works on PTB (Step 8), scales up cleanly (Step 9), and transfers to three other domains (Step 10). Let's close the loop on the running sentence, then check you can re-explain the whole idea.
Back to “the cat that chased the mouse … was ___.” With the recurrent memory kept clean, the subject “cat” reaches the final position intact, so the model puts most of its probability on a fitting predicate. Here's its next-word distribution:
The top guesses all fit a cat as the subject. Had dropout corrupted the recurrent path, “cat” would be gone by this point and the distribution would flatten into noise. (Probabilities are toy values for our running example; the qualitative behaviour is the paper's point.)
Now you can explain it. Five questions — answer each out loud before opening it. If all five come easily, you've genuinely got this paper.
What is the paper's one-line fix?
Apply the dropout operator only to the non-recurrent (vertical, layer-to-layer) connections, and never to the recurrent (horizontal, timestep-to-timestep) connections. In the equation, wrap only hl−1t in D(·); leave hlt−1 bare.
Why did plain dropout break LSTMs?
On the recurrent path a memory passes the same dropout point once per timestep, so the corruption accumulates with sequence length. Long-range dependencies — the whole reason LSTMs exist — get wiped before they can be used.
What is the L+1 rule?
With dropout confined to the vertical wires, a memory is corrupted exactly L+1 times on its way to a prediction — L for the layers plus 1 for the output — independent of how many timesteps it travelled. Constant corruption, unbounded memory.
What did it buy on Penn Tree Bank?
A non-regularized LSTM scored 114.5 test perplexity. Regularizing let the model grow much wider: medium (650u) hit 82.7, large (1500u) hit 78.4, and a 38-model ensemble reached 68.7. Lower perplexity is better.
Is the trick LSTM-specific?
No. The same placement rule applies to any recurrent net, and the paper confirmed it on speech recognition, machine translation, and image captioning — improving the held-out metric in every one. It addresses recurrence itself, not one task.
What happened next — this gave RNN practitioners a reliable way to regularize, and the idea was refined into variational/recurrent dropout, zoneout, and (alongside) layer normalization. Transformers later sidestepped the recurrent-noise problem entirely by dropping recurrence — but this is the clean origin of “where exactly to put dropout in a recurrent net.”
“By not using dropout on the recurrent connections, the LSTM can benefit from dropout regularization without sacrificing its valuable memorization ability.” — §3.2, Zaremba, Sutskever & Vinyals, 2014
A one-symbol rule that made recurrent nets safe to scale.
By pinning down exactly where dropout belongs in a recurrent net, this paper turned a “dropout doesn't work here” folklore into a reliable recipe — and that recipe became the starting point for a wave of better recurrent regularizers.
Variational & recurrent dropout
Later work (Gal & Ghahramani; Semeniuta et al.) extended noise onto the recurrent path safely by fixing one mask across timesteps — building directly on this placement insight.
Zoneout · LayerNorm
Zoneout randomly keeps memory unchanged instead of zeroing it; layer normalization steadied recurrent activations. Both share the goal of regularizing RNNs without wrecking memory.
Transformers
Attention models dropped recurrence altogether, sidestepping recurrent noise — but they still lean on plain dropout, placed with the same care this paper taught.
One operator, placed deliberately — and big recurrent nets stopped overfitting their way out of usefulness.