A VAE stores each bit wherever it's cheapest — so design the decoder to make global structure the only thing it can't store on its own.
A VAE (variational autoencoder — a generative
model that encodes data into a small latent code z, then
decodes it back) has a well-known embarrassment: pair it with a strong
autoregressive decoder and the code z gets
ignored entirely — it collapses to noise. People blamed
"optimization challenges." This paper's quiet brilliance is to stop
treating that as a bug.
The authors give a bits-back argument for why the code
is ignored: a VAE is secretly a two-part code, and any bit it can
reconstruct more cheaply with the decoder alone will never be paid for
twice by also storing it in z. Then they weaponize the
principle — shrink the decoder's window so local texture is cheap to
model pixel-by-pixel, and the code is left holding exactly the
global structure the decoder can no longer see.
"Ignored latent code" stops being a failure and becomes a dial: choose what the decoder can model locally, and you choose what the code abstracts away.
The puzzle
A code that goes silent — and a bits-back reason why
Step 1 · The black box & the running example
A box that compresses an image — on purpose, lossily.
Here is the whole story in one picture. A 28×28 handwritten digit goes
in. The box squeezes it down to a tiny latent code z
— about 19.2 bits for a binarized MNIST image — and then
decompresses a fresh image back out. The catch: the output is
not pixel-perfect. The digit's identity and overall shape survive,
but the fine texture — stroke width, exactly which pixels are on — is
redrawn. This single digit is our running example
for all nine steps.
Why would anyone want a lossy autoencoder? Because in many tasks the useful part of an image is its global content — the object, the shape, the identity — and the fine texture is noise you'd rather throw away. VLAE lets you decide, by architecture, exactly which kind of information the code keeps and which it discards. That control is the prize, and it set new likelihood records for VAEs along the way.
From the abstract: “by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only ‘autoencodes’ data in a lossy fashion.”
Step 2 · What a VAE is, in one diagram
Encode to a code, decode back, pay for both.
You've seen the box compress a digit lossily. Before we ask why the code can go silent, we need the three moving parts of a VAE — they're the cast of the rest of the page.
A VAE has three pieces. The encoder q(z|x) reads the
digit and proposes a latent code z (a short vector). The
decoder p(x|z) reads that code and tries to redraw the
digit. And a prior p(z) says what codes are "normal" —
in a vanilla VAE, a plain Gaussian. Training maximizes a single
objective, the ELBO (evidence lower bound —
a tractable lower bound on the data's log-likelihood).
Encoder q(z|x)
Reads the digit → a short code z.
Decoder p(x|z)
Reads the code z → redraws the digit.
The ELBO objective
Reconstruct well, but pay a KL price for every bit you put in z.
ℒ(x)=E[log p(x|z)]−DKL(q(z|x)‖p(z))
left term = reconstruct the digit well · right term = the toll you pay for the bits you stored in z
That second term is a cost on using the code. Every bit you push
into z raises the KL toll. So if the decoder can reconstruct
some piece of the digit without the code, paying to store it in
z is pure waste — the model will avoid it. Step 3 turns that
into a real failure.
ELBO = evidence lower bound. Maximizing it both fits the data and keeps the code distribution close to the prior. In 2D images the decoder is usually a simple factorized distribution p(x|z) = ∏ᵢ p(xᵢ|z) that tends to reconstruct the input sharply.
Step 3 · The silent-code complaint
Give the decoder enough power and the code goes silent.
Step 2 left us with a cost on the code. Now make the decoder strong and watch that cost decide the code's fate.
Swap the simple factorized decoder for a powerful
autoregressive one — a PixelCNN or RNN that
predicts each pixel from the pixels already drawn,
p(x) = ∏ᵢ p(xᵢ | x<ᵢ). Such a decoder is a universal
density estimator: it can model the whole image by itself,
with no help from z. And when it can, the well-documented
result is posterior collapse — the code is completely ignored
and the model regresses to a plain unconditional autoregressive net.
Weak decoder · the code is used
The decoder can't model the digit alone, so the code must carry the content — a big, healthy KL term.
Strong autoregressive decoder · code ignored
The decoder models the digit by itself, so storing anything in z is wasted KL — the code collapses to noise.
The literature usually blames "optimization challenges": early in
training the encoder carries little about x, so the model
sets the posterior to the prior to dodge the KL cost, and never
recovers. The authors' key observation is sharper — this is
not just an optimization quirk. Even with the optimizer solved
perfectly, for most realistic VAEs with a powerful enough decoder, the
code should still be ignored at the optimum.
If the code is ignored even at the perfect optimum, then it is not a bug to be fixed by better training — it is a property to be understood and steered. Step 4 names the principle (bits-back), and Step 5 turns it into a knob.
Posterior collapse: KL(q(z|x)‖p(z)) → 0, so q(z|x) ≈ p(z) and the code carries no information about x. Documented in detail by Bowman et al. (2015) for RNN decoders in sequence modeling.
The fix
Bits-back · shrink the window · an autoregressive prior for free
Step 4 · The bits-back two-part code
A bit lives wherever it is cheapest to store.
Step 3 claimed the code should be ignored at the optimum, not just during training. Here is the accounting that proves it — and it's a coding-theory idea, not an optimization one.
Think of a VAE as a way to transmit a digit using a two-part
message. First send the code z — the "essence" — using a
number of bits set by the prior. Then send the correction: how
the real digit differs from what z alone implies, using the
decoder p(x|z). Bits-back coding
(an information-theoretic view of variational inference) shows the
expected message length of this scheme equals exactly the
negative ELBO. Minimizing message length is maximizing
the ELBO.
Now the punchline. If the decoder can model some piece of the digit
locally, on its own, then encoding that piece into z
would only pay the bits twice. A length-minimizing coder never does that.
So there is an information preference:
modelled locally by p(x|z)→stays out of z
only the remainder — what the decoder can't reach locally — is encoded in z
On statically binarized MNIST, the VLAE's converged code carries 13.3 nats = 19.2 bits per image. An identical VAE with an ordinary factorized decoder spends 37.3 bits. Same data, roughly half the code — the code learned a lossier, more global summary because the decoder absorbed the rest locally.
1 nat = 1.4427 bits (log₂ e). The extra coding cost of an imperfect posterior is exactly D_KL(q(z|x)‖p(z|x)) nats — always ≥ 0 — so for any realistic model that gap won't vanish, and the cheap-to-model bits stay out of z.
Step 5 · Shrink the decoder's window
Make global structure the one thing the decoder can't see.
Step 4 said cheap-to-model-locally bits stay out of the code. So if we control what's modellable locally, we control what the code keeps. That control is a single architectural choice.
The PixelCNN decoder predicts each pixel from a
receptive field — a small window of nearby,
already-drawn pixels. Make that window small. Now local texture
(stroke width, edges) is cheap to model from the immediate neighbors and
stays out of z. But anything long-range — the digit's
overall shape, which strokes connect to which — falls outside
the window. The decoder can't reach it locally, so by information
preference it must be encoded in the code. You've forced
z to hold exactly the global structure.
Drag the decoder's window — watch the local / global bit split move
Where each bit goes
Small window → the code must remember most of the structure: a rich, global code.
Slide right and the decoder's window grows: more of the image becomes locally predictable, so the code's job shrinks toward holding only the roughest shape. Slide left to 1×1 and the decoder is blind — the code must carry almost everything. The window size is the representation knob.
Formally, the decoder factorizes over a windowed context
p(xᵢ | xWindowAround(i)) instead of the full
history x<ᵢ. As long as the window is smaller than the
full context, the decoder can't represent arbitrary long-range
dependencies on its own — so those are pushed into z.
Nothing here fights the information-preference property — it uses it. By deciding what the decoder can and cannot model locally, you decide what gets abstracted into the code. Global structure for 2D images is just one choice; a heavily down-sampled window would instead keep long-range patterns in the decoder and push high-frequency detail into the code.
Receptive field xWindowAround(i): the small block of already-generated pixels a pixel may depend on. The paper's decoder is a 6-layer masked-convolution PixelCNN with filter size 3, so the dependency window is a small local patch.
Step 6 · What the code keeps vs drops
Identity stays; texture is regenerated.
You set the window in Step 5 to push global structure into the code. Now decompress and look: the running-example digit comes back with its identity intact but its texture replaced.
To inspect the lossy code, the authors take a test image, encode it to
z ~ q(z|x), then sample a fresh image
x′ ~ p(x|z) — a "decompression." Across MNIST these
decompressions are never exact reconstructions: the global shape
and digit identity are preserved, but the binary mask is usually
different and local style — stroke width — shifts. The code threw the
texture away; the decoder invented new, plausible texture.
Original test digit
A seven, drawn with a thick stroke and a crossbar.
Decompression x′ ~ p(x|z)
Same seven, same outline — thinner stroke, new pixels. Identity kept, texture regenerated.
The code doesn't always keep the kind of global information you had in mind — it depends on the decoder constraint. On OMNIGLOT, whose characters vary more inside small patches, some decompressions don't preserve the exact symbol. The lesson the authors draw: you must specify what to keep by designing the decoder for the dataset and task.
Their CIFAR-10 study makes the knob explicit. As the receptive field grows, the decoder captures more structure on its own, so the code is left holding less:
| Decoder window | What the code z keeps |
|---|---|
| 4×2 · small | detailed shape — the code still carries fine outline |
| 5×3 · medium | shape, less detail — texture handled locally |
| 7×4 · large | only the rough shape — the decoder does the rest |
| 7×4 grayscale window | rough shape plus colour — colour can't be modelled in grayscale, so it's forced into z |
One slider does it all. A grayscale-only window can't predict colour locally, so colour gets pushed into the code — a different lossy summary from the very same machinery. What to abstract away becomes an explicit architectural decision, not a hope.
Figures 1–3 of the paper. The decompressions are real samples, not reconstructions: z keeps global content, p(x|z) regenerates everything the window can model on its own.
Step 7 · A better prior, for free
An autoregressive flow prior buys expressiveness at no extra cost.
The windowed decoder (Steps 5–6) decides what the code keeps. The paper's second improvement makes the whole model a tighter density estimator — and it's a quietly elegant free lunch.
The vanilla prior p(z) is a plain Gaussian — often too
simple to match the codes the encoder actually produces, which wastes
bits. The fix is to make the prior learnable with an
autoregressive flow (AF) — an invertible map
z = f(ε) that turns simple Gaussian noise ε
into a flexible code distribution. The authors show this AF prior is
mathematically equivalent to placing an inverse-autoregressive-flow
(IAF) posterior on the noise — the same expressiveness gain people
already used on the encoder side.
Tested in isolation — same network, just an AF prior instead of an equivalent IAF posterior, no autoregressive decoder — on statically binarized MNIST it reduces test NLL by 0.6 nat (train by 0.8 nat). The deeper generative path is the source of the gain, and it costs nothing at training time.
A VLAE = a windowed PixelCNN decoder (Steps 5–6, controls the representation) plus an AF prior (this step, tightens the bound). One gives you the lossy code you want; the other makes the model a record-setting density estimator. Step 8 reads the scoreboard.
Autoregressive flow: z = f(ε) with log p(z) = log u(ε) + log|dε/dz|. Because AF prior and IAF posterior have identical cost under z ~ q(z|x), the more expressive generator is genuinely free.
The payoff
New likelihood records — and can you now explain it?
Step 8 · The scoreboard
Lossy on purpose — and a record density estimator.
You've built the whole machine: the silent-code puzzle (Steps 1–3), bits-back (Step 4), the windowed decoder knob (Steps 5–6), the AF prior (Step 7). Did the combination actually win?
Density estimation is measured in NLL — negative log-likelihood, the number of nats it takes to describe a held-out image; lower is better. With a single set of hyperparameters tuned on static MNIST, the same VLAE set new state-of-the-art among latent-variable models on three binary datasets at once.
Report card · static MNIST · test NLL in nats (lower is better)
VLAE · AF prior + PixelCNN
PixelRNN · van den Oord 2016
AF VAE · no AR decoder
IAF VAE · Kingma 2016
Discrete VAE · Rolfe 2016
78test NLL nats · lower is better82
Read the report card and the headline holds twice over. The code is deliberately lossy — it keeps global structure and throws away texture — yet the full model is a more accurate density estimator than the strong unconditional decoder alone. Controllability and likelihood improved together, not at each other's expense.
NLL is estimated with importance sampling (4096 samples for the binary datasets, 512 for CIFAR-10). "Unconditional Decoder" is the same PixelCNN trained with no VAE part — the honest baseline for what the code adds.
Step 9 · Now you can explain it
Five questions — say each answer out loud first.
You've walked the whole argument: the silent code, the bits-back reason, the window knob, the free prior, the scoreboard. Time to check that it stuck.
Answer each out loud before opening it. If all five come easily, you've genuinely got this paper.
Why does a strong autoregressive decoder make the code go silent?
Because a powerful decoder can model the whole image by itself, and the ELBO charges a KL toll for every bit stored in z. Storing what the decoder already predicts would pay twice, so a length-minimizing model leaves the code empty — posterior collapse. It happens even at the optimum, not just early in training.
What is the information-preference principle?
In a VAE viewed as a bits-back two-part code, any information the decoder can model locally without z is encoded locally; only the remainder goes into z. So whatever the decoder can't reach on its own is exactly what the code is forced to keep.
How do you force the code to keep global structure?
Shrink the PixelCNN decoder's receptive field. Local texture becomes cheap to model from nearby pixels and stays out of z, while long-range structure (the digit's shape) falls outside the window and must be encoded in z. On MNIST that lossy code is ~19.2 bits vs 37.3 for a regular VAE.
What's the AF prior, and why is it "for free"?
A learnable autoregressive-flow prior z = f(ε) that replaces the plain Gaussian. It's equivalent to an IAF posterior but with a deeper generative path, and the cost under z ~ q(z|x) is identical — so the extra expressiveness costs nothing at training time. It cut static-MNIST test NLL by 0.6 nat on its own.
Is the lossy code a sacrifice in likelihood?
No. The code is deliberately lossy, yet the full VLAE is a better density estimator than the strong unconditional decoder alone (78.53 vs 87.55 nats on dynamic MNIST) and set records on three binary datasets. Controllability and likelihood improved together.
What happened next — VLAE's deepest legacy is conceptual: it remains the cleanest statement of information preference, the reason expressive decoders starve their latent space. The pattern — split a model into a global latent and a local autoregressive part — recurs in PixelVAE, hierarchical VAEs, VQ-VAE, and the latent stage of modern latent-diffusion image generators.
“By designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only ‘autoencodes’ data in a lossy fashion.” — the abstract, Chen et al., 2016
The reasoning outlived the benchmarks.
Diffusion models and giant autoregressive transformers long since passed these likelihood numbers. What endured is the idea: a clean explanation of why expressive decoders ignore their latents, and the recipe for splitting a model into a global code and a local autoregressive part.
Why latents collapse
The bits-back account reframed posterior collapse from an optimization bug into a coding fact. It is still the cleanest answer to "why doesn't my VAE use its latent space?"
Global code + local model
Separating global structure (a latent) from local detail (an autoregressive or diffusion decoder) became a standard pattern — PixelVAE, VQ-VAE, hierarchical VAEs, and the latent stage of latent-diffusion image generators.
Lossy by design
Deciding what a representation should discard — by shaping what the decoder can model on its own — turned "what to abstract away" into an explicit architectural dial for representation learning.
Stop fighting the silent code — design the decoder, and the code keeps exactly what you want.