The famous complaint about VAEs paired with strong autoregressive decoders is that they "ignore" the latent code entirely — the decoder is expressive enough to model the data on its own, so the code collapses to noise. This paper's quiet brilliance is to stop treating that as a bug. It gives a bits-back argument for why the code gets ignored (the decoder will only use latents for information that is cheaper to store there than to model autoregressively), and then weaponizes the principle: shape the decoder so that local detail is cheap to model locally, and the latent is left holding exactly the global structure you wanted.
Key Findings
- Lossy coding is a feature, not a failure. By capping the PixelCNN decoder's receptive field, local texture becomes cheap to reconstruct pixel-by-pixel, so the latent code is forced to encode only what the decoder can't see locally — global structure. Reconstructions keep the shape and identity while swapping the fine texture.
- A bits-back account of latent collapse. The "information preference" argument explains posterior collapse not as an optimization quirk but as the model rationally storing each bit wherever it is cheapest, which makes latent usage something you can engineer rather than hope for.
- Autoregressive flows as the prior. Using autoregressive models for both the prior p(z) and the decoder p(x|z) tightened the bound enough to set new likelihood records on MNIST, Omniglot and Caltech-101 Silhouettes, with competitive CIFAR-10 numbers.
- Receptive field as a representation knob. What the code learns is dialed by the decoder's window size, turning "what to abstract away" into an explicit architectural decision.
Great Fit / When to Skip
Great fit if you want the conceptual root of why expressive decoders starve the latent space, or a principled way to control what a VAE abstracts versus reconstructs — it remains the cleanest statement of the information-preference idea. Look elsewhere if you want today's image generators; diffusion and large autoregressive transformers long since overtook these likelihood numbers, and the lasting value here is the reasoning, not the benchmark.
