Most latent generative pipelines train a reconstruction-focused tokenizer (e.g., VAE/autoencoder) and then freeze it for a downstream prior. This two-stage design can yield latents that reconstruct well but are poorly organized for generation. GenFirst reframes the trade-off: generation objectives should shape latent geometry first, and reconstruction should be applied later as a refinement. That change mitigates latent collapse and the generation–reconstruction conflict that plague direct end-to-end training.
Key Findings
- Entropy term in KL is critical to avoiding latent collapse: reconstruction and prior-fitting pressures both tend to shrink posterior uncertainty; keeping an entropy term preserves non-degenerate latent uncertainty so latents remain usable for generation.
- Asymmetric learning dynamics: reconstruction is fast and strongly supervised, while fitting a prior (generation) is slower and harder. Joint objectives without scheduling cause reconstruction to dominate and collapse generative utility.
- GenFirst strategy: prioritize a generative objective under weak reconstruction pressure to first shape a generation-friendly latent manifold; then progressively increase reconstruction weight to recover high-frequency visual detail. This simple schedule stabilizes end-to-end training without needing to freeze components.
- Empirical validation: tested with continuous autoregressive priors and SiT priors (implicit likelihoods). Reported results include SiT achieving gFID 0.97 with classifier-free guidance and 1.45 without guidance on ImageNet-256, and MMDiT reaching a GenEval score of 0.90 on text-to-image generation.
- Generality: the approach extends beyond image-only generation to shared visual latents for representation learning and to continuous unified text–image generation setups, indicating the strategy targets a general optimization failure mode rather than a single architecture.
What Sets It Apart
- Focus on optimization dynamics rather than architectural fixes: instead of designing a different tokenizer or a specialized prior, GenFirst changes the training schedule to let the slower generative objective shape the manifold before reconstruction tightens it.
- Explicit role for entropy regularization: highlights that KL's entropy component is not a mere technicality but a practical lever to keep latents informative during joint training.
- Practical, minimal intervention: GenFirst is a scheduling idea that can be applied with different priors (exact-likelihood autoregressive priors and implicit SiT priors were tested) and thus integrates with diverse latent-generation stacks.
Who it's for and tradeoffs
Great fit if you are training or researching latent-space generative models and want end-to-end learning without freezing tokenizers—especially when you care about sample quality from learned priors rather than only reconstruction fidelity. It helps when you can modify training schedules and have access to both generative and reconstruction losses.
Look elsewhere if your pipeline must prioritize perfect reconstruction (e.g., certain compression tasks) or if you cannot run the extra experiments needed to tune generation-first schedules and entropy regularization. The method requires careful balancing/scheduling of loss weights and may interact with the choice of prior; gains depend on appropriate priors and compute for training the slower generative objective.
Where it fits
GenFirst complements work that studies what makes tokenizer/latent manifolds generation-friendly (e.g., prior-aligned tokenizers). Instead of only reshaping latent geometry via auxiliary regularizers, GenFirst alters the optimization path so that the generative objective has a dominant early role. Combineable with tokenizer-level regularizations for further gains.