A straight line through time, with valves on it.
A plain recurrent network (RNN — a network that reads a sequence one step at a time, feeding its own output back in as memory) keeps rewriting its memory at every word. Useful signal from far back gets squashed step after step, so by the time it's needed it has faded — the long-term dependency problem. The fix is the LSTM (Long Short-Term Memory).
The LSTM adds one new wire: the cell state, a horizontal line running straight across the top of every diagram, touched only by a couple of cheap linear operations. Information can ride it nearly unchanged for as long as you like — Olah's image is a conveyor belt. Three gates (each just a sigmoid that outputs a number between 0 and 1) sit on the belt and decide, at every step, what to throw off, what to load on, and what to read out.
It is the uninterrupted belt — not the gates themselves — that lets gradients flow back over long gaps without being repeatedly squashed.
The problem
Why a plain RNN forgets — and what it forgets
Step 1 · The black box & the running example
A box that reads a sentence one word at a time.
Here is the whole job in one picture. We're building a language model — a box that reads a sentence left to right and, at each word, guesses the next one. To do that it must remember what it already read. Our running example for the entire page is one sentence:
“Jane grew up in France. … She speaks fluent ____”
To fill the blank with French, the box has to recall a word that appeared far earlier. To get the pronoun She right, it has to remember that the subject — Jane — is female. That single fact, “the gender of the current subject,” is the piece of memory we will track through all ten steps.
This is the explainer everyone secretly relies on. Its contribution isn't a new model — the LSTM is from Hochreiter & Schmidhuber, 1997. It's a single picture that made the equations finally click for a generation of practitioners: the cell state as a belt, the gates as valves. The rest of this page rebuilds that picture, one gate at a time.
From the post: “Humans don't start their thinking from scratch every second. … Traditional neural networks can't do this, and it seems like a major shortcoming.” The whole point of a recurrent net is to carry something forward.
Step 2 · What a plain RNN does
A loop that rewrites its whole memory every word.
You've met the box and the sentence it must remember. Open it up just enough to see the simplest version — a plain RNN — and why its memory is fragile.
An RNN is a network with a loop. At each step it takes the new word
xt and its previous memory
ht−1 (the hidden state, a vector that
summarizes everything so far), mashes them through one tanh
layer, and the result becomes both the output and the new memory
ht. Unrolled, it's a chain — one copy of the
same cell per word.
That single layer is the catch. Each step squashes the whole memory through a tanh and a weight matrix. A fact written down early gets re-squashed at every later word, so its influence shrinks multiplicatively — the basis of the vanishing-gradient problem (Hochreiter, 1991; Bengio et al., 1994).
A plain RNN has no separate, protected place to store a fact. Its one memory vector is both the working scratchpad and the long-term store, and it's rewritten every step. The LSTM's whole trick (Part B) is to add a second, lightly-touched line just for keeping things.
Step 3 · The long-term dependency problem
Short gaps are easy; long gaps are where it breaks.
Step 2 showed the memory is one vector, rewritten every word. Now see exactly when that fails — and the post's two example sentences make it crisp.
Sometimes the clue is near the blank. To finish “the clouds are in the ____,” you only need the last few words — a plain RNN handles that fine. But to finish “I grew up in France … I speak fluent ____,” the clue (France) sits far back. As the post puts it, as that gap grows, “RNNs become unable to learn to connect the information.”
Short gap · plain RNN copes
“the clouds are in the sky” — the clue is right there.
Long gap · plain RNN forgets
“I grew up in France … I speak fluent ____” — the clue is far away.
In theory a plain RNN could learn to carry “France” all the way; in practice the optimization makes long-range carrying very hard. The signal has to survive being re-squashed at every intervening word, and it doesn't.
So the requirement is sharp: we need a path along which a fact can travel many steps with almost no distortion. Not a better squashing layer — a way to mostly not squash. That is the single idea the LSTM is built around.
The problem isn't that the network lacks memory — it's that its only memory gets overwritten and re-squashed every step. Give it a second line that is barely touched, and a fact can ride it across a long gap intact. Part B builds exactly that line.
The “clouds … sky” and “France … French” examples are the post's own. LSTMs were introduced to attack this gap by Hochreiter & Schmidhuber, 1997, and refined by many since.
The gated belt
The cell state · what a gate is · forget · input · update · output
Step 4 · The cell state, the conveyor belt
A second line that runs straight across the top.
Step 3 said we need a path a fact can ride for many steps without being squashed. Here it is — the one wire that makes an LSTM an LSTM.
The cell state Ct is the horizontal line
running straight across the top of the diagram. Olah's exact words:
“It runs straight down the entire chain, with only some minor linear
interactions. It's very easy for information to just flow along it
unchanged.” That's the conveyor belt. Below it sits the rest of
the cell, deciding what to do to the belt.
In a plain RNN the memory is pushed through a full tanh layer every step (Step 2). Here the memory rides a line touched only by a multiply and an add — both nearly linear. The gradient flows backward along it without being repeatedly squashed. That uninterrupted path, not the gates, is what defeats vanishing gradients.
The two operations on the belt are pointwise — they act on each number in the vector independently. We'll meet the machinery that produces them in Steps 6–9.
Step 5 · What a gate is
A gate is a dial from 0 to 1, multiplied in.
You've seen a multiply and an add touch the belt. The multiplier is always a gate — and once you see what a gate is, all three of them become the same simple thing.
A gate is, in Olah's words, “a way to optionally let information
through.” It is one sigmoid layer — the function
σ that squashes any number into the range 0 … 1 — followed
by a pointwise multiply. The sigmoid's output g is
the dial: “A value of zero means ‘let nothing through,’ while a value
of one means ‘let everything through.’” To gate a number
v, you just compute g · v.
Drag the gate from 0 to 1 — watch how much of v survives the multiply
At g = 0.10, only 0.08 of the value 0.80 survives — almost forgotten. Push g to 1 and the full 0.80 passes; push to 0 and nothing does.
This one operation is all three gates. A forget gate multiplies the old belt; an input gate scales new candidates before adding them; an output gate scales what gets read out. Same dial, three jobs.
Now the formula arrives as a receipt for the dial you just
dragged. A gate g looks at the previous output
ht−1 and the new word xt,
and produces its 0 … 1 value with a sigmoid:
gt=σ(W·[ht−1,xt]+b)
σ = sigmoid, squashes into 0 … 1 · [h, x] = the two inputs stacked together · W, b = this gate's learned weights · every gate below has this exact shape
The forget gate (ft), input gate
(it) and output gate (ot)
are the same equation with their own W and b. Learn this one
shape and you've learned all three — the only difference is which part
of the belt each one multiplies.
σ(z) = 1 / (1 + e−z). The sigmoid never quite reaches 0 or 1, so a gate “mostly” keeps or drops — which is exactly what lets it learn smooth, partial decisions.
Step 6 · The forget gate
Decide what to throw off the belt.
You know what a gate is (Step 5): a 0 … 1 dial multiplied in. The first gate in the cell points that dial at the old belt and asks: what should we drop?
The forget gate ft looks at
ht−1 and xt and outputs
a number in 0 … 1 for each number in the cell state — “1
represents ‘completely keep this’ while a 0 represents ‘completely get
rid of this.’” In our language model, the belt holds the gender of
the current subject. While we're still talking about Jane, keep it.
The moment a new subject appears, forget the old gender so a new
one can take its place.
ft=σ(Wf·[ht−1,xt]+bf)
same gate template as Step 5 · its job: multiply the old belt Cₜ₋₁ by fₜ
Step through the sentence — watch the forget gate decide what stays on the belt
Reading word Jane — step 1 of 4.
Start of the sentence: nothing to forget yet — the gate sits open at f = 1.0, the belt passes through unchanged.
Step to “Bob” (a new subject) and watch the gate slam toward f = 0: the old gender is multiplied to ≈ 0 — dropped from the belt — exactly so the input gate (next step) can write the new one. The forget gate is how the LSTM makes room.
From the post: “the cell state might include the gender of the present subject, so that the correct pronouns can be used. When we see a new subject, we want to forget the gender of the old subject.” The forget gate is that decision, made fresh at every word.
fₜ is a vector — one 0 … 1 value per number in the cell state — so the cell can keep some facts while dropping others in the same step. We show a single “gender” number for clarity.
Step 7 · The input gate & the candidate
Decide what new thing to load onto the belt.
The forget gate cleared space (Step 6). Now the cell decides what to write into that space — this takes two little networks working together.
First, an input gate it (a sigmoid, 0 …
1) decides which values are worth updating. Second, a
tanh layer proposes a vector of candidate values
C̃t — the new content that could be
added (tanh squashes into −1 … 1, so a candidate can push the belt
positive or negative). In the language model: when Jane is replaced by
Bob, the candidate is “male,” and the input gate opens to let it in.
it= σ(Wi·[ht−1,xt]+bi)
C̃t= tanh(WC·[ht−1,xt]+bC)
iₜ = how much to write (a gate) · C̃ₜ = what to write (the candidate) · the next step combines them
Input gate iₜ · how much
A sigmoid in 0 … 1 — a dial on each candidate number. Near 0, the candidate barely lands on the belt; near 1, it's written in full. It decides which updates matter right now.
Candidate C̃ₜ · what
A tanh in −1 … 1 — the proposed new content. For our sentence, “the new subject is male” might be +1; “female” might be −1. The gate chooses how strongly to commit it.
From the post: “we'd want to add the gender of the new subject to the cell state, to replace the old one we're forgetting.” Two gates, two jobs: forget cleared the old (Step 6), input proposes and admits the new (here).
Why split it into a gate and a candidate? So the network can decide what to store and how much of it independently — more expressive than one combined layer.
Step 8 · Update the belt
Forget the old, add the new — one line of arithmetic.
You have a forget gate (Step 6) and an input gate + candidate (Step 7). This is the step where they actually act on the belt — and you can do the arithmetic by hand.
Updating the cell state is two operations the belt already showed in
Step 4: multiply the old state by the forget gate, then
add the input-gated candidate. In the post's words: “We multiply
the old state by ft, forgetting the things we
decided to forget earlier. Then we add it ·
C̃t.”
Ct=ft·Ct−1+it·C̃t
forget the old belt + load the new candidate = the updated belt
Let's do the moment Jane → Bob by hand, on the single “gender” number.
Old belt Ct−1 = +1.0 (female, say). A new
subject arrives, so the forget gate closes:
ft = 0.1. The candidate is “male,”
C̃t = −1.0, and the input gate opens,
it = 0.9. Then:
Ct= (0.1)(+1.0) + (0.9)(−1.0) =−0.8
old female memory (+1.0) is wiped to +0.1 · new male memory (−0.9) is written in · belt flips to −0.8
The belt went from +1.0 (female) to −0.8 (now mostly male) in one multiply-and-add — no big nonlinear layer in the way. From the post: “this is where we'd actually drop the information about the old subject's gender and add the new information.” Everything else is in service of this one line.
Numbers are a toy 1-D illustration; real cell states are long vectors and each number updates the same way. The arithmetic Cₜ = f·Cₜ₋₁ + i·C̃ₜ is exactly the paper's.
Step 9 · The output gate
Decide what to read off the belt right now.
The belt is now updated (Step 8). The cell still has to produce an
actual output for this step — the hidden state ht
that feeds the prediction and the next cell. That's the third gate.
The output gate ot (a sigmoid, 0 … 1)
decides which parts of the belt to expose. The belt is first run
through a tanh (to land it in −1 … 1), then multiplied by the
gate: ht = ot · tanh(Ct).
The full memory keeps riding the belt; only a filtered view of it
leaves the cell.
On our sentence: having just seen a subject, the cell “might want to output information relevant to a verb” — for example, whether the subject is singular or plural, so the next word's verb is conjugated correctly. It reads out exactly the part of memory the next word needs, and keeps the rest.
ot= σ(Wo·[ht−1,xt]+bo)
ht=ot· tanh(Ct)
oₜ = how much of the belt to reveal · tanh(Cₜ) = the belt, squashed to −1 … 1 · hₜ = the cell's output for this step
Forget (drop), input + candidate (add), update (the belt arithmetic), output (read out). The memory keeps riding the belt undisturbed; hₜ is only a filtered view of it. That separation — a protected store plus a controlled readout — is the entire LSTM.
From the post: “since it just saw a subject, it might want to output information relevant to a verb … whether the subject is singular or plural, so that we know what form a verb should be conjugated into.”
The payoff
Variants, what it powered — and can you now explain it?
Step 10 · Variants & recap
Same belt, fewer gates — and can you now explain it?
You've assembled the standard LSTM (Steps 4–9). The post closes by noting it's one point in a design space — small rewirings of the same belt-and-gates idea.
Two common variants. Peephole connections let the gates also
look at the cell state itself, not just ht−1 and
xt — a richer view of memory. The GRU
(Gated Recurrent Unit, Cho et al., 2014) goes the other way: it
merges the forget and input gates into a single “update gate,”
and merges the cell and hidden states into one. Fewer parts, often
comparable results.
| Variant | Gates | What changes |
|---|---|---|
| Standard LSTM | forget · input · output | the cell you just built — separate cell & hidden states |
| Peephole | 3, each sees Cₜ | gates also peek at the cell state for sharper timing |
| GRU | update · reset | forget + input merged; cell & hidden states merged — simpler |
The post's verdict: the variants are mostly minor. What matters is the shared idea — a lightly-touched memory line regulated by sigmoid gates. As Olah notes, “essentially all” the striking results people were getting from recurrent nets in 2015 were achieved with LSTMs.
Now you can explain it. Five questions — answer each out loud before opening it. If all five come easily, you've genuinely got this.
Why does a plain RNN forget long-range facts?
Its only memory is one hidden-state vector pushed through a tanh layer every step (Step 2). A fact written early is re-squashed at every later word, so its influence and its gradient shrink multiplicatively — vanishing gradients. There's no protected place to keep something.
What actually defeats vanishing gradients in an LSTM?
The cell state — the conveyor belt. It runs straight across with only a pointwise multiply and add touching it (Step 4), so a fact, and the gradient, can ride it almost unchanged over many steps. As the post stresses, it's the uninterrupted belt, not the gates, that does this.
What is a gate, in one sentence?
A sigmoid that outputs a number in 0 … 1, multiplied pointwise into some vector — 0 lets nothing through, 1 lets everything through (Step 5). All three gates are this same operation with their own weights.
Walk the cell-state update by hand.
Cₜ = fₜ·Cₜ₋₁ + iₜ·C̃ₜ. In Step 8: old belt +1.0 (female), forget gate f = 0.1 wipes it to +0.1; candidate −1.0 (male) with input gate i = 0.9 adds −0.9; the belt becomes (0.1)(+1.0) + (0.9)(−1.0) = −0.8 — the subject flipped.
What does the output gate change — and not change?
hₜ = oₜ · tanh(Cₜ) exposes only a filtered view of the belt as this step's output (Step 9). The full memory stays on the belt, untouched, and rides on to the next step. The cell can keep a fact while choosing not to reveal it yet.
How is a GRU different from an LSTM?
A GRU merges the forget and input gates into one update gate and merges the cell and hidden states into one — fewer parameters, same belt-and-gates spirit, often comparable accuracy.
What happened next — LSTMs powered the first wave of usable machine translation, speech recognition and image captioning (the seq2seq era, 2014–2017). The post even points at what comes after: attention, letting a step look back over a whole collection of past states — the seed of the Transformer, which this post predates.
“It runs straight down the entire chain, with only some minor linear interactions. It's very easy for information to just flow along it unchanged.” — Christopher Olah, on the cell state, 2015
The belt taught a field how to remember.
For nearly a decade, the LSTM was sequence learning — and this post is how most people came to understand it. Its two lasting gifts: a mechanism that carried the first wave of real-world language and speech AI, and a way of explaining that made the equations finally obvious.
Translation, speech, captions
Stacked LSTMs powered the first usable neural machine translation, speech recognition and image captioning. For years, “essentially all” impressive recurrent-net results, as the post says, were LSTMs.
The road to the Transformer
The post's own closing note points past LSTMs to attention — letting a step look over all past states at once. That idea became the Transformer (2017), which retired recurrence and now powers modern LLMs.
A way of explaining
The conveyor-belt-and-valves picture set a standard for visual ML teaching. The same author's later work (the Illustrated Transformer lineage it inspired) still shapes how the field draws its ideas.
One lightly-touched line through time — and a network could finally keep a thought.