The most persuasive argument for sequence models was never a benchmark — it was a screen full of fake Shakespeare, fake C code, and fake math papers produced by a model that had only ever been told to guess the next character. Karpathy's bet here is pedagogical: instead of explaining what RNNs can do, he shows you their raw output and lets the strangeness of it do the convincing. The lasting lesson is that "predict the next token" is a deceptively far-reaching training objective — the same idea that, scaled up, became the entire premise of modern language models.
Core Idea
- Character-level, not word-level. The model sees one character at a time and predicts the next, so it has to discover spelling, punctuation, brackets, and indentation from scratch — making its competence feel earned rather than templated.
- Output is the proof. Samples trained on Shakespeare, the Linux kernel source, Wikipedia markup, and LaTeX read as eerily plausible, demonstrating that an RNN captures structure across long spans, not just local n-grams.
- Neurons are interpretable. Visualizing individual cells reveals ones that track quote-open/quote-close state or position inside a line — concrete evidence that the hidden state stores meaningful, human-legible features.
- Simplicity is the point. A short, hackable training loop produces these results, which is why "char-rnn" became a rite of passage and the on-ramp many people took into deep learning.
Who It Fits / When to Skip
Great fit if you want intuition for why next-token prediction works, or a vivid first contact with sequence modeling before touching attention. Look elsewhere if you need a current architecture — RNNs and LSTMs were largely displaced by the Transformer, so treat this as the conceptual seed of that lineage, not a recipe for today's models.
