Before this paper, the prevailing wisdom was that dropout and recurrent networks simply did not mix — applying the most effective regularizer of the era to an LSTM made it worse, not better. The fix turned out to be almost embarrassingly small: dropout was being sprayed onto the recurrent connections, repeatedly corrupting the cell's memory at every timestep so it could never carry information across long spans. Confine the noise to the non-recurrent (vertical, layer-to-layer) connections and leave the recurrent (horizontal, timestep-to-timestep) path clean, and the contradiction disappears.
Key Findings
- The recurrent path must stay noiseless. Dropping units on the timestep-to-timestep connection accumulates corruption proportional to sequence length, destroying long-term memory; restricting dropout to between-layer connections preserves it.
- Bigger RNNs become trainable. With overfitting controlled, the authors could scale LSTMs far past previously practical sizes and still see gains rather than memorization.
- It generalized across modalities. The same rule improved language modeling, speech recognition, machine translation, and image caption generation — a sign it addressed something structural, not task-specific.
- Regularization without forgetting. The model still memorized the long-range dependencies LSTMs exist to capture, so the gain in generalization came at no cost to capacity.
Why It Still Matters / When to Skip
Great fit if you want the clean origin of "where exactly to put dropout in a recurrent net," or a compact case study in how a tiny architectural distinction unlocks scale. Look elsewhere if you need today's recurrent regularizers — variational/recurrent dropout, zoneout, and layer normalization refined this idea, and Transformers later sidestepped the recurrent-noise problem entirely. This is the foundational insight, not the final word.
