Variational Lossy Autoencoder

Reframes the VAE's tendency to ignore its latent code as a controllable design choice: by limiting a PixelCNN decoder's receptive field and using autoregressive flow priors, the code is forced to keep only global structure and discard local texture.

Visual Explainer Visit Website

Introduction

The famous complaint about VAEs paired with strong autoregressive decoders is that they "ignore" the latent code entirely — the decoder is expressive enough to model the data on its own, so the code collapses to noise. This paper's quiet brilliance is to stop treating that as a bug. It gives a bits-back argument for why the code gets ignored (the decoder will only use latents for information that is cheaper to store there than to model autoregressively), and then weaponizes the principle: shape the decoder so that local detail is cheap to model locally, and the latent is left holding exactly the global structure you wanted.

Key Findings

Lossy coding is a feature, not a failure. By capping the PixelCNN decoder's receptive field, local texture becomes cheap to reconstruct pixel-by-pixel, so the latent code is forced to encode only what the decoder can't see locally — global structure. Reconstructions keep the shape and identity while swapping the fine texture.
A bits-back account of latent collapse. The "information preference" argument explains posterior collapse not as an optimization quirk but as the model rationally storing each bit wherever it is cheapest, which makes latent usage something you can engineer rather than hope for.
Autoregressive flows as the prior. Using autoregressive models for both the prior p(z) and the decoder p(x|z) tightened the bound enough to set new likelihood records on MNIST, Omniglot and Caltech-101 Silhouettes, with competitive CIFAR-10 numbers.
Receptive field as a representation knob. What the code learns is dialed by the decoder's window size, turning "what to abstract away" into an explicit architectural decision.

Great Fit / When to Skip

Great fit if you want the conceptual root of why expressive decoders starve the latent space, or a principled way to control what a VAE abstracts versus reconstructs — it remains the cleanest statement of the information-preference idea. Look elsewhere if you want today's image generators; diffusion and large autoregressive transformers long since overtook these likelihood numbers, and the lasting value here is the reasoning, not the benchmark.

Back

Information

Websitearxiv.org
OrganizationsOpenAI, UC Berkeley
AuthorsXi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel
Published date2016/11/08

More Items

Large Language Model Papers2026

Multi-Stream Attention Residuals

Xiaohan Qin, Xiangdong Zhang +8School of AI, Shanghai Jiao Tong University, Shanghai Innovation Institute +3

Enables multi-head, depth-wise residual retrieval in Transformers by expanding a single residual stream into multiple context-aware streams using causal depthwise convolutions. Improves pretraining convergence and downstream benchmark scores with minimal overhead, yielding notable iso-loss compute speedups in large MoE models.

paper transformers mixture-of-experts moe depth+3

Large Language Model Papers2025

Foundations of Large Language Models

Tong Xiao, Jingbo ZhuNLP Lab, Northeastern University, NiuTrans Research

A concise textbook-style book that explains foundational concepts and techniques for large language models, covering pre-training, generative models, prompting, alignment, inference, and reasoning. Structured as self-contained chapters for readers with some ML/NLP background or those seeking a principled introduction to LLM foundations.

foundation-model LLM NLP paper book+5

Large Language Model Papers2026

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Jinyuan Li, Chengsong Huang +5Washington University in St. Louis, University of Michigan, Ann Arbor

Analyzes why self-evolving reasoning models collapse under repeated self-training and proposes R-Quest: a feedback-driven pipeline that trains solvers to reject invalid questions and uses a frozen base model to detect task-level repetition, filtering training data to sustain multi-round gains.

reasoning paper llm benchmarks math+2