Long before "regularization" meant dropout or weight decay tuned by a grid search, this 1993 paper framed overfitting as a coding problem: a network generalizes only when its weights carry less information than the outputs they explain. That reframing is the quiet payload here — not a benchmark win, but a way of thinking. The "bits-back" argument it introduces, where the apparent cost of transmitting noisy weights is partly refunded, would resurface two decades later as the backbone of variational autoencoders and modern variational inference.
Key Findings
- Weights as a message to be compressed. Generalization is recast through the Minimum Description Length principle: keep the description length of the weights small, and you control complexity directly rather than through ad hoc penalties.
- Noise as the regularizer. Adding adaptive Gaussian noise to each weight makes "information content" a differentiable quantity, so the trade-off between squared error and weight information can be optimized by gradient descent, not guesswork.
- The bits-back insight. Sending stochastic weights looks wasteful, but the receiver can recover the random bits used to pick them — a refund that makes the true coding cost match a variational bound, the same identity VAEs later exploit.
- Exact, cheap derivatives. With linear output units the relevant gradients compute in closed form, sidestepping the Monte Carlo estimates such stochastic schemes usually demand.
Why It Still Matters / When to Skip
Great fit if you want to trace where variational inference in deep learning actually began, or to see the bits-back coding argument stated cleanly before it became VAE folklore. Look elsewhere if you want empirical muscle — the experiments are preliminary, showing only a slight edge over weight decay; the lasting value is conceptual, the lineage it started, not the numbers it reported.
