"The best explanation of the data is the one that compresses it most." That single sentence is the entire bet of the Minimum Description Length principle, and this tutorial's quiet achievement is making it precise enough to actually compute with. Where Occam's razor is a slogan and Kolmogorov complexity is beautiful but uncomputable, MDL carves out the practical middle: a way to measure model complexity in bits that you can write down and minimize.
Key Findings
- Learning is compression. Fitting a model is reframed as finding the shortest two-part code — the bits to describe the model plus the bits to describe the data given that model. Overfitting becomes visible as a code that got longer, not just a number that drifted.
- Complexity gets a unit. Instead of hand-waving about "simpler" models, MDL assigns model classes a concrete description length, so a flexible class pays for its flexibility in bits and a fair comparison falls out automatically.
- Two passes on purpose. The first chapter is deliberately non-technical — intuition with no formulas — and the second makes every one of those intuitions mathematically exact. The structure itself is the teaching method.
- A bridge, not a slogan. It connects the philosophical (Occam, induction) to the operational (universal codes, normalized maximum likelihood), which is why MDL spread across statistics and machine learning rather than staying a curiosity.
How It Works
The core trick is replacing "probability of the data" with "length of the shortest code for the data." Because there is a tight correspondence between codes and probability distributions, choosing the model that yields the shortest total code length is a principled, prior-free way to balance fit against complexity — no arbitrary regularization constant required.
Who Fits / When to Skip
Great fit if you want the conceptual and formal foundations of MDL from one of the principle's clearest expositors, or a rigorous lens on why simpler models generalize. Look elsewhere if you need ready-to-run model-selection recipes for a specific deep-learning pipeline — this is the principle and its mathematics, not a modern practitioner's cookbook.
