“Guess the next character” is a deceptively far-reaching goal.
The most persuasive argument for sequence models was never a benchmark. It was a screen full of fake Shakespeare, fake C code and fake math papers, produced by a model that had only ever been told one thing: given the characters so far, guess the next one. No grammar rules, no dictionary, no notion of words — just a stream of characters and a single prediction repeated millions of times.
Karpathy's bet here is pedagogical. Instead of explaining what a recurrent neural network (RNN — a net with a memory that carries information from one step to the next) can do, he shows you its raw output and lets the strangeness do the convincing. The lasting lesson is that to predict the next character well, the model is forced to discover spelling, brackets, indentation and quotes on its own — and that the very same “predict the next token” objective, scaled up, became the entire premise of modern language models.
Train a tiny memory to do one thing — guess the next character — and structure you never taught it falls out for free.
The machine from orbit
What is this thing, and why does it need a memory?
Step 1 · The black box
Characters go in, one more character comes out — repeat.
Here is the whole story in one picture. You feed the model the
characters it has seen so far — say hell — and it returns
its best guess at the single next character: o.
Append that guess, feed the longer string back in, and you have a text
generator. This four-letter toy — hello over the tiny
vocabulary [h, e, l, o] — is the paper's own example, and
it will follow us through every step:
That is the entire job. The astonishing part is what happens when you train this loop on a big pile of text and then let it run free. Below is a genuine sample from a model trained only on the works of Shakespeare (4.4 MB of plays) — never told what a word, a line break or a speaker name is:
From the post: “There's something magical about Recurrent Neural Networks… I've now trained RNNs for a while and I'm still amazed by how well they work.”
Step 2 · Why a plain network won't do
Text has no fixed length — so the model needs a memory.
You've seen the loop run and the eerie output it produces. Before we open the box, one question: why use a recurrent net at all, instead of an ordinary one?
An ordinary (“feed-forward”) network has a rigid contract: a fixed-size input in, a fixed-size output out, in a fixed number of steps. That's fine for a photo, but text is a variable-length sequence — “hi” and a thousand-page novel are both valid inputs, and the next character can depend on something thousands of steps back. There's no fixed box that fits.
The RNN's answer is a memory it carries from one character to
the next: a small vector called the hidden
state h. At each step it folds the new character
into h, then reuses the same network on the
next character. One compact net, applied over and over, can read a
sequence of any length.
h, the only path by which the past reaches the present.The hidden state is the model's entire knowledge of the past. Everything the RNN knows about the characters it has already read has to be squeezed into that one running vector — which is exactly why it's interesting to ask, later, what each number in it learns to track.
From the post: an RNN's API lets it “operate over sequences of vectors” — sequence in, sequence out — where a plain net is locked to fixed-size vectors. Training a plain net is optimization over functions; training a recurrent net is optimization over programs.
Step 3 · Inside the cell
One step does only three things — and updates the memory.
We said the RNN reuses one small network at every character, carrying
a memory h forward. Let's open that one box and see the
three moves it makes.
At each step the cell takes two things — the new character
x and the old memory h — and produces two
things: a fresh memory and a list of scores for what comes next.
Here is the whole computation, exactly as the post gives it in three
lines of numpy:
The whole cell, in code and in one line
Read it as: ht = tanh(Whh ht−1 + Wxh xt), then yt = Why ht.
tanh just squashes every number into the range −1…1 so
the memory can't blow up. That is the entire forward pass — the
three moves in the picture, written twice.
The output scores y depend on h, and
h depends on every character seen so far. So the
guess for the next character is shaped by the whole history — not just
the one character you just typed. (In practice the paper uses an LSTM,
a fancier cell with the same shape but a better-behaved memory.)
That's the machine. Everything from here on is about feeding characters into this cell, reading the scores out, and asking what the memory learned to store.
Guessing the next character
Make the quizzes · score by hand · unroll the memory · sample text
Step 4 · Turn text into practice questions
One word, “hello”, becomes four next-character quizzes.
We have a cell that, given the characters so far, emits scores for the next one. To train it, we need a pile of questions whose answer we already know. Text supplies them for free.
Take the training word hello. Its vocabulary has just four
distinct characters: [h, e, l, o]. Walking left to right,
every prefix is a question and the character that follows is the
answer — so this one word is four labelled examples:
hel the next char is l, but in hell it is o. It must use the memory h to know where in the word it is.
One more nicety. A character isn't a number the net can multiply, so
each one becomes a one-hot vector — a list
that is all zeros except a single 1 at that character's slot. With
four characters, h = [1,0,0,0], e = [0,1,0,0],
and so on. That single 1 is the x fed into the cell.
“Predict the next character” needs no labels and no rules — any text is its own answer key. That is the whole reason the idea scales: point it at 4 MB of Shakespeare or 474 MB of Linux source and the training examples write themselves.
The post's exact framing: feed hell in, and the model
should output high scores for e, l, l, o at the four steps
— “the same character l has different targets, so the RNN cannot rely on
the input alone and must use its recurrent connection to keep track of
the context.”
Step 5 · Pass 2 — compute it by hand
Six small steps. Real numbers. No magic left.
We have the four quizzes. Now watch the cell answer the very first
one — given h, predict the next character — with the
exact numbers from the post, so there's nothing left to take on faith.
Feed in h. The cell's output layer hands back one raw
score (a logit) for each of the four
possible next characters. In the post's worked example those scores
are:
The correct answer is e (because the word is
hello). But the model's biggest score is on
o — it's wrong. Three little moves turn these scores into
probabilities and then into a single number that says how
wrong:
Exponentiate — make every score positive
Softmax turns a list of scores into probabilities. First raise e to each score, which kills the negatives and stretches the big ones apart.
Divide by the total — now they sum to 1
Add the four up (2.7 + 9.0 + 0.05 + 60.3 = 72.1) and divide each by that total. The result is a probability for every next character.
Score the answer — the loss
Training only cares about the probability on the correct
character, e. The loss is
−ln(p): it is 0 when the model is certain and right,
and grows as p shrinks. Here p = 0.125,
so the loss is large.
Training nudges every weight a hair in the direction that would raise the green score (e) and lower the red ones (h, l, o). Repeat across millions of characters and the model learns to put its probability where the next character actually tends to be. Drag the scores and feel softmax react:
Drag a score — the green one (e) is the answer we want to win
Loss on the correct character “e”: 2.08. Raise e's score (or lower o's) and watch P(e) climb toward 1 and the loss fall toward 0 — that is exactly what one training step tries to do.
This is the entire learning signal. No grammar, no rules — just “raise the probability of the character that actually came next.” Everything the model ever learns about English, code or LaTeX is squeezed in through this one tiny pressure, applied billions of times.
The logits 1.0 / 2.2 / −3.0 / 4.1 are the post's own example (“it assigned confidence of 1.0 to h, 2.2 to e, −3.0 to l, and 4.1 to o”). We'd want the green number high and the red numbers low.
Step 6 · Unroll it across a word
The memory is how the two “l”s end up with different answers.
You scored one character. Now run the cell across the whole word
hell, carrying the memory forward — and watch the puzzle
from Step 4 resolve itself.
At each step the same cell takes the new character and the
memory from the step before. By the time it reaches the third
character, the memory has quietly recorded “I'm two characters into
hel…”; one step later it knows “now I'm at
hell…”. Same input letter l, different
memory — so different prediction:
l only because the memory carries where it is in the word.
A net with no memory would be stuck: it sees l and must
give one fixed answer. The memory breaks that tie — and the same trick
scales up to remembering an open quote 40 characters ago, or which
function you're inside. We'll see exactly that in Step 10.
The post: because the same character “l” has different targets, “the RNN cannot rely on the input alone and must use its recurrent connection to keep track of the context to achieve this task.”
Step 7 · Sampling — and the temperature dial
To write, you don't pick the top character — you roll the dice.
Training is done; the model can score the next character. To generate text, we run the Step 1 loop: get the probabilities, pick a character, feed it back, repeat. The only question is how we pick.
Always taking the single highest-probability character makes the
model loop and repeat itself. Instead we sample — draw a
character at random, weighted by its probability. One knob,
temperature, reshapes the odds before the
draw: low temperature sharpens them (play it safe), high temperature
flattens them (take risks). Say the model has just read
th in English text — drag the dial:
Drag temperature — low = conservative & repetitive, high = creative & error-prone
T = 0.70 · balanced. “e” leads (the, they, their…) but the others keep a real chance, so the text stays varied without falling apart.
Near T → 0 the post's model gets stuck repeating its single most likely continuation (“…the same thing that was a startup…” over and over). Crank it up and it grows adventurous but starts inventing non-words. The sweet spot in between reads as fluent, surprising text.
Temperature divides every score before softmax. The six probabilities
here are recomputed live from a fixed toy distribution over
e, a, i, o, r, space; the qualitative effect is exactly the
one the post demonstrates on Paul-Graham-trained samples.
What it actually learns
The output gallery · what the neurons track · how it gets there
Step 8 · The output gallery
Point the same loop at different text, get a different forger.
You now understand the entire machine: read characters, score the next one, sample, repeat. The post's payoff is to run that one recipe on wildly different training sets and show the raw results.
Nothing about the model changes between these — same char-RNN, same objective. Only the training text differs, and with it the structure the model is forced to learn:
Linux kernel C474 MB of source · 3-layer LSTM · ~10M params
LaTeX algebraic geometry16 MB book source · multi-layer LSTM
Wikipedia markup~96 MB (Hutter Prize) · overnight LSTM
MARN_EPT and indicate_policy are invented, and some variables are declared but never used.The model nails local, structural rules — matching brackets, indentation, valid tags — because those depend on nearby characters. It fumbles long, semantic ones — closing the right LaTeX environment 30 lines later, or making the sentence actually mean something. The memory is good, but finite.
The post also trains on 8,000 baby names and gets plausible new ones — “Rudi, Levette, Berice, Lussa, Mareanne” — of which about 90% never appear in the training set. It isn't copying; it's modelling.
Step 9 · Looking inside the memory
A few neurons quietly learned to track quotes, lines and code depth.
We keep saying “the memory tracks where you are.” The most striking part of the post is that you can open up that memory and literally watch it do so — one number at a time.
Recall that h is a vector of numbers (a neuron
is just one of those numbers). Karpathy colours each character by a
single chosen neuron's value — red for low, green for high — and a
handful turn out to be cleanly interpretable. One example: a neuron
that switches on inside quotation marks and off outside,
even across long stretches:
The post catalogues several of these: a cell that tracks position
inside a line (rising steadily, useful for line breaks), one that
lights up only inside URLs, one that counts how far into
www it is, and one that flips on inside
[[ … ]] markdown. About 5% of neurons are this
crisply interpretable; the rest are a tangle.
The single objective “guess the next character” forced the model, with no further hints, to invent useful book-keeping features — quote state, line position, bracket depth. That emergence is, in Karpathy's words, one of the cleanest examples of where the power of end-to-end training comes from.
Most neurons aren't interpretable — they fire in messy, distributed ways. The interpretable few are evidence, not the whole story: a hint that the hidden state stores meaningful, human-legible structure.
Step 10 · Learning, in stages
It learns spaces first, then words, then long-range structure.
We've seen the finished forger and peeked inside its memory. The last question: how does an empty model get there? The post snapshots the Shakespeare model at different points in training.
Early on the loss is high and the samples are garbage. As the single pressure from Step 5 — “raise the probability of the character that came next” — is applied over and over, structure appears in a remarkably consistent order: first the rhythm of words and spaces, then short common words, then spelling and quotes, and only much later the long-range stuff:
Spaces and short words are local — they only need the last few characters, so they're learned first. Quotes, names and structure need the memory to span longer gaps, so they show up later. The model climbs the same ladder of dependency length we walked from Step 6 to Step 9.
The post stresses there's no curriculum forcing this order — it falls out of plain gradient descent on the one next-character objective. Easy, short-range patterns reduce the loss fastest, so they're learned first.
Step 11 · The score sheet & recap
A tiny model, a one-line objective — and can you now explain it?
You've watched it read, score, sample, generate, and learn in stages. The post's argument was never a leaderboard — it was the sheer implausibility of these results from so little machinery.
Score sheet · what one char-RNN pulled off in 2015
structure learned from characters alone — no rules, no labels
Now you can explain it. Six questions — answer each out loud before opening it. If all six come easily, you've genuinely got this blog post.
Why does the RNN need a memory at all?
Text is a variable-length sequence and the next character can
depend on something far back. A fixed-size plain net can't take a
sequence; the hidden state h lets one small net read
any length, carrying the past forward (Step 2).
In “hello”, why can't it just memorize l → l?
Because hel is followed by l but
hell by o — same input letter, different
answer. Only the memory, which records how far into the word it
is, tells the two apart (Steps 4 and 6).
What exactly does softmax + the loss do?
Softmax turns the raw scores into probabilities that sum to 1;
the loss −ln(p) on the correct character is the only
training signal. You computed it by hand: logits 1.0 / 2.2 / −3.0
/ 4.1 → P(e) = 12.5% → loss 2.08 (Step 5).
What does temperature change?
It rescales the scores before softmax. Low temperature sharpens the distribution (safe, repetitive text); high temperature flattens it (diverse but error-prone). It changes how you sample, not what the model knows (Step 7).
Why does it nail brackets but flub long LaTeX?
Matching brackets and indentation are local — a few characters of memory suffice. Closing the right environment 30 lines later needs the memory to span a long gap, which is where a finite hidden state strains (Step 8).
What's the evidence the memory stores real structure?
You can read it out. About 5% of neurons turn out to track human concepts — quote-open/close state, position in a line, depth inside brackets — with nobody having programmed them (Step 9).
What happened next — “predict the next token” went from a cute demo to the entire training objective of GPT and every large language model. The char-RNN you just walked through is the conceptual seed of that whole lineage.
“There's something magical about Recurrent Neural Networks. … We'll train RNNs to generate text character by character and ponder the question ‘how is that even possible?’” — Andrej Karpathy, 2015
“Predict the next token” turned out to be everything.
The specific architecture aged — RNNs and LSTMs were largely displaced by the Transformer. But the idea this post made vivid, that next-token prediction is a deceptively powerful objective, only grew.
char-rnn
The hackable ~100-line training loop became a rite of passage — the way a generation first got their hands dirty with deep learning.
GPT & the LLMs
Swap the RNN cell for a Transformer and scale the same next-token objective — and you get the models behind today's chat assistants.
Reading the neurons
Watching individual cells learn quotes and brackets foreshadowed the whole field of mechanistic interpretability — opening the box of what a trained net knows.
A model told only to guess the next character taught us that the objective, not the cleverness, was the point.